CMAQv5.0.2 Readme file
Release Date: April 2014
Contents
Overview
This README file outlines the steps necessary to build and run the CMAQ version 5.0.2 (CMAQv5.0.2) release. The build and run scripts that are included in the tar files are set up to compile and run on Linux (see Testing Notes). In case you want to run on another platform, the C-shell scripts are easy to modify for other Unix implementations.
The CMAQv5.0.2 release is available to the community accompanied by example scripts that invoke a specific configuration of the model. This configuration has been used by U.S. EPA in unit tests, system tests, and for model evaluation prior to the release of the model. There are other features and options within the CMAQ modeling system in this release, beyond this specific model configuration, that have not yet been fully tested, evaluated or documented, that are also available to users. As always, we are interested in feedback from the community on experiences using the CMAQ modeling system.
The CMAQv5.0.2 distribution package includes:
- Base model package with the accompanying libraries and all programs required for building and running CMAQ.
- EPA-developed instrumented versions of CMAQ with only the CCTM codes, scripts, and configuration files
- Community-contributed versions of CMAQ with only the CCTM codes, scripts, and configuration files
- Note that users should install and build the base model first to create the full suite of CMAQ programs and libraries before installing the instrumented and/or community packages.
Base Model
- CMAQv5.0.2.Apr2014.tar.gz - gzipped tar file containing source code for models, tools, and libraries, and C-Shell scripts to build and execute the CMAQ model
Instrumented models
- CMAQv5.0.2_DDM3D.Apr2014.tar.gz - gzipped tar file containing source code for CMAQ-DDM3D model version
- CMAQv5.0.2_ISAM.Apr2014.tar.gz - gzipped tar file containing source code for CMAQ-ISAM model version
- CMAQv5.0.2_STM.Apr2014.tar.gz - gzipped tar file containing source code for CMAQ-STM model version
Community Contributions
- CMAQv5.0.2_VBS.Apr2014.tar.gz - gzipped tar file containing source code for community-contributed versions of the model
NOTE: You must have I/O API version 3.1 and netCDF
Build Instructions
The following outlines a sequence of steps you can follow to build the base CMAQ model
Install the CMAQv5.0.2 source codes and scripts
In a working directory, gunzip and untar the model distribution tar file, CMAQv5.0.2.Apr2014.tar.gz. This will produce the following subdirectories:
CMAQv5.0.2/
models/
scripts/
Configure and source config.cmaq
Edit the file CMAQv5.0.2/scripts/config.cmaq to set the environment variable for M3HOME, and to setup your computer system information, including compilers and compiler flags. Environment variables M3MODEL and M3LIB are automatically set in the config.cmaq file. Source the config file to set up the needed environment for building and running CMAQ.
source $M3HOME/scripts/config.cmaq
Install test data
cd to $M3HOME and gunzip and untar the data tar file
DATA.CMAQv5.0.2.Apr2014.tar.gz
First download the file from this directory on the google drive:
CMAQv5.0.2 Directory
Or download directly from here DATA.CMAQv5.0.2.Apr2014.tar.gz
tar -xzvf DATA.CMAQv5.0.2.Apr2014.tar.gz
will produce the following subdirectories:
data/
bcon/ <<<<<<< empty, to be filled by the user
cctm/ <<<<<<< empty, to be filled by the user
crop/
dust/
emis/
icon/ <<<<<<< empty, to be filled by the user
jproc/ <<<<<<< empty
lightning/
mcip/
ocean/
procan/
raw/
phot/
Install required external libraries
When you source the config.cmaq script the directory $M3HOME/lib ($M3LIB) will be created
netCDF
The CMAQ build scripts look for the netCDF library and include files in the $M3LIB path as $M3LIB/netcdf. If netCDF is installed elsewhere on your system, create a symbolic link in $M3LIB/netcdf to the existing netCDF.
Example: cd $M3LIB ln -s /share/linux86_64/wdx/lib/x86_64i/ifc/netcdf-3.6.2 netcdf
I/O API
CMAQv5.0.2 requires I/O API version 3.1, download and install this library on your system following the instructions provided by the I/O API web site (http://cmascenter.org/ioapi/). The CMAQ build scripts look for the I/O API library in the $M3LIB path as $M3LIB/ioapi_3.1. If I/O API is installed elsewhere on your system, create a symbolic link in $M3LIB/ioapi_3.1.
Example: cd $M3LIB/ioapi_3.1 ln -s /share/linux86_64/wdx/lib/x86_64i/ifc/ioapi_31/Linux2_x86_64ifort . ln -s /share/linux86_64/wdx/lib/x86_64i/ifc/ioapi_31/ioapi .
mpich
If you are running a multiprocessor application, then the "mpich" library and include files are required to build CMAQ. CMAQ has been successfully built and run with mvapich1, mvapich2, intel_mpi, and openmpi. The CMAQ build scripts look for the mpich library and include files in the $M3LIB path as $M3LIB/mpich. If mpich is installed elsewhere on your system, create a symbolic link in $M3LIB/mpich.
Example: cd $M3LIB ln -s /share/linux86_64/wdx/lib/x86_64i/ifc/mpich mpich
Build CMAQ libraries and bldmake
Create the stencil exchange library required for parallel processing (se_snl) and serial processing (sef90_noop):
cd $M3HOME/scripts/stenex Execute (type) bldit.se Execute (type) bldit.se_noop
For parallel CCTM operation create the parallel I/O library (pario):
cd $M3HOME/scripts/pario Execute (type) bldit.pario
Create bldmake, the tool required to build the executables for the CMAQ processors, model and tools.
cd $M3HOME/scripts/build execute (type) bldit.bldmake
- Note: Although bldmake is really a tool, it gets installed under $M3LIB
Build CMAQ executables
Create the model executables: ICON and BCON need to be compiled and run separately for profile data (coarse grid) and for nest data (fine grid); CCTM is compiled only once. If you are using the inline photolysis module in the CCTM, then you will not need to run JPROC. Follow the following steps to build the CMAQ executables for the test simulation:
cd $M3HOME/scripts/icon ./bldit.icon |& tee bldit.icon.profile.log cd $M3HOME/scripts/bcon ./bldit.bcon |& tee bldit.bcon.profile.log cd $M3HOME/scripts/cctm ./bldit.cctm |& tee bldit.cctm.log
Run Instructions
General Instructions
The CMAQv5.0.2 distribution package includes a full set of test input data needed to run the model for one day. The general approach for running CMAQ includes these steps, in this order:
- Prepare WRF meteorology data with MCIP
- Run ICON and BCON to create initial and boundary conditions input data
- Run the CCTM to estimate air quality fields
These steps assume that the user has available WRF meteorology and SMOKE emissions data.
Although MCIP meteorology data are provided with the CMAQv5.0.2 test data, MCIP4.2 code and scripts are packaged in the CMAQv5.0.2 distribution. Users can compile and run MCIP to process WRF meteorology data for input to CMAQ and SMOKE. See the SMOKE.pdf readme file included with the CMAQv5.0.2 documentation for discussion on preparing emissions inputs for CMAQ.
ICON
For the CMAQ test case, start with ICON (cd to $M3HOME/scripts/icon). Run the script "run.icon" to create CMAQ-ready initial conditions (ICs) file. These data files will be written to the output directory defined in the ICON run script ($M3DATA/icon). This script will produce profile-based ICs for a 12km CCTM simulation.
- Note: It's always a good idea to capture in a log file the text written to standard out when running these models. In each "run" script, near the top, is a suggested method (e.g. for ICON):
run.icon >&! icon.log &
Check the ICON log file to ensure complete and correct execution.
BCON
After running ICON, change directories to $M3HOME/scripts/bcon and run the command "run.bcon >&! bcon.log &" to create a BC file for CMAQ. The default BCON run script also produces the profile-based BC dataset for a 12km CCTM simulation. BCON may also be configured to produced BCs for a nested simulation using output concentrations from the CCTM. See the CMAQv5x Operational Guidance Document for a discussion on creating inputs for nested simulations.
CCTM
Next run the CCTM executable. Change directories to $M3HOME/scripts/cctm and run the command "run.cctm >&! cctm.log &". By default, the CCTM runscript is setup for parallel processing. Users will likely need to configure the script for the configuration needed to submit parallel processing jobs on their systems.
Following completion of the CCTM run, you should have a complete collection of datasets which you can compare with the distribution datasets in DATA_REF.CMAQv5.0.2.Apr2014.tar.gz. Unless you modify the run scripts, the output data from all the models will be output to the following paths:
$M3DATA/
bcon/
cctm/
icon/
Other details
- You can check CMAQ output file headers (and data) using the netCDF utility ncdump.
- The GRIDDESC file written by MCIP contains horizontal projection and grid domain definitions that are required input for many CMAQ models. The CMAQ run scripts include an environment variable that points to the GRIDDESC file.
- The horizontal grid definition can be set to window from the met and emissions input files. However, the window must be a "proper subset" (i.e., a subset from the interior of the domain and not including boundaries). Note: The domains represented by the met and emissions data must be the same.
- Running CCTM for a windowed domain or a higher resolution nested domain from larger or coarser met and emissions datasets requires creating initial and boundary data for the target domain.
Testing Procedures
| Compiler | netCDF | I/O API | MPI_YN (#P) | MPI | Benchmark Timing (HH:MM:SS) |
|---|---|---|---|---|---|
| Intel Fortran version 11.1.059 | |||||
| Gfortran version 4.8.1 | 4.1.1 | 3.1 (04/09/12) | N | N/A | |
| Gfortran version 4.8.1 | 4.1.1 | 3.1 (04/09/12) | Y (16) | mvapich2-1.7 | 0:36:36 |
| Intel Fortran version 14.0.0 | 4.1.1 | 3.1 (04/09/12) | N | N/A | 6:34:03 |
| Intel Fortran version 14.0.0 | 4.1.1 | 3.1 (04/09/12) | Y (16) | mvapich2-1.7 | 0:39:01 |
| Intel Fortran version 14.0.0 | 4.1.1 | 3.1 (04/09/12) | Y (16) | openMPI-1.4.2 | 0:39:01 |
| PG Fortran version 10.3 | 4.1.1 | 3.1 (04/09/12) | N | N/A | 4:43:24 |
| PG Fortran version 10.3 | 4.1.1 | 3.1 (04/09/12) | Y (16) | openMPI-1.4.2 | 0:46:58 |
References
Contact
Shawn Roselle and Jeff Young, Atmospheric Modeling and Analysis Division, U.S. EPA
