CMAQv5.1 Readme file
Release Date: November 2015
- 1 Overview
- 2 Build Instructions
- 3 Run Instructions
- 4 Testing Procedures
- 5 References
- 6 Contact
This README file outlines the steps necessary to build and run the CMAQ version 5.1 (CMAQv5.1) release. The build and run scripts that are included in the tar files are set up to compile and run on Linux (see Testing Notes). To run CMAQv5.1 on another hardware platform, the C-shell scripts can be adapted for other Unix implementations.
The CMAQv5.1 release is available to the community accompanied by example scripts that invoke a specific configuration of the model. This configuration has been used by U.S. EPA and the CMAS Center in unit tests, system tests, and for model evaluation prior to the release of the model. There are other features and options within the CMAQ modeling system in this release that have not yet been fully tested, evaluated or documented. Feedback from the community on experiences using the CMAQ modeling system is welcome and can be provided via email to the contacts listed below.
The CMAQv5.1 distribution package includes:
- The base model and scripts package with the CMAQ source code, preprocessors, and utility programs
- Input data for a 12-km tutorial/benchmark CCTM simulation
- Reference output data for verifying the CMAQ installation
- CMAQv5.1.Nov2015.tar.gz - gzipped tar file containing source code for models, tools, and libraries, and C-Shell scripts to build and execute CMAQv5.1.
- Anticipated release in 2016
NOTE: You must have I/O API version 3.1 and netCDF
The following outlines a sequence of steps to build the base suite of CMAQ software, including the CMAQ Chemical Transport Model (CCTM).
Install the CMAQv5.1 source codes and scripts
In a working directory, gunzip and untar the model distribution tar file, CMAQv5.1.Nov2015.tar.gz. This will produce the following subdirectories:
CMAQv5.1/ models/ scripts/
Configure and source config.cmaq
Edit the file CMAQv5.1/scripts/config.cmaq to set the environment variable for M3HOME, and to set up your computer system information, including compilers and compiler flags. Environment variables M3MODEL and M3LIB are automatically set in the config.cmaq file. Navigate to the CMAQv5.1/scripts directory and use the following command to set the environment variables in the config file to set up the needed environment for building and running CMAQ.
cd CMAQv5.1/scripts source config.cmaq
Install test data
cd to $M3HOME and gunzip and untar the data tar file, DATA.CMAQv5.1.Nov2015.tar.gz. This will produce the following subdirectories:
data/ bcon/ bidi/ crop/ dust/ emis/ icon/ lightning/ mcip/ ocean/ phot/ procan/
Install required external libraries
When you source the config.cmaq script the directory $M3HOME/lib ($M3LIB) will be created.
The CMAQ build scripts require the netCDF library and include files in the $M3LIB path as $M3LIB/netcdf. If netCDF is installed elsewhere on your system, create a symbolic link in $M3LIB/netcdf to the existing netCDF.
Example: cd $M3LIB ln -s /usr/local/lib/x86_64/ifc-13.1/netcdf-4.3.0 netcdf
CMAQv5.1 requires I/O API version 3.1, download and install this library on your system following the instructions provided by the I/O API web site (http://cmascenter.org/ioapi/). The CMAQ build scripts look for the I/O API library in the $M3LIB path as $M3LIB/ioapi. If I/O API is installed elsewhere on your system, create symbolic links in $M3LIB/ioapi.
Example to link for I/O API version 3.1: cd $M3LIB mkdir ioapi cd ioapi ln -s /usr/local/lib/x86_64/ifc-13.1/ioapi_3.1/Linux2_x86_64ifort lib ln -s /usr/local/lib/x86_64/ifc-13.1/ioapi_3.1/Linux2_x86_64ifort include ln -s /usr/local/lib/x86_64/ifc-13.1/ioapi_3.1/ioapi/fixed_src src
To run CMAQv5.1 across multiple processors, the "mpich" library and include files are required to build CMAQ. CMAQv5.1 has been successfully built and run with mvapich1, mvapich2, intel_mpi, and openmpi. The CMAQ build scripts look for the mpich library and include files in the $M3LIB path as $M3LIB/mpi. If mpich is installed elsewhere, create a symbolic link in $M3LIB/mpi.
Example: cd $M3LIB ln -s /usr/local/lib/x86_64/ifc-13.1/mpich mpi
Build bldmake utility
Create bldmake, the tool required to build the executables for the CMAQ, model and supporting software and tools.
cd $M3HOME/scripts/build bldit.bldmake
- Note: Utility bldmake will be installed under $M3LIB
Build CMAQ executables
Create the model executables: The initial conditions (ICs) preprocessor ICON and boundary conditions (BCs) preprocessor BCON need to be compiled and run separately to generate ICs/BCs from profile data or to generate ICs/BCs for a nested domain. Follow these steps to build the CMAQ executables:
cd $M3HOME/scripts/icon ./bldit.icon |& tee bldit.icon.profile.log cd $M3HOME/scripts/bcon ./bldit.bcon |& tee bldit.bcon.profile.log cd $M3HOME/scripts/cctm ./bldit.cctm |& tee bldit.cctm.log
The meteorology preprocessor MCIP uses a Makefile rather than bldit scripts. Navigate to the MCIP source code directory and configure the Makefile for your Fortran compiler. Run the command "make" to build the MCIP executable
cd $M3HOME/scripts/mcip/src [edit and save the Makefile] make |& tee make.mcip.log
The CMAQv5.1 distribution package includes a full set of test input data needed to run the CCTM for one day. Instructions are provided below for the general approach for running CMAQ and for running the CMAQ tutorial case, which only requires running the CCTM.
General Simulation Instructions
These steps assume that the user has available WRF meteorology and SMOKE emissions data.
- Run MCIP to prepare the CMAQ-ready meteorological data from the WRF output
- Run ICON and BCON to create initial and boundary conditions input data
- Run the CCTM to estimate air quality fields
Although MCIP meteorology data are provided with the CMAQv5.1 test data, MCIP4.3 code and scripts are packaged in the CMAQv5.1 distribution. Users can compile and run MCIP to process WRF meteorology data for input to CMAQ and SMOKE.
Default initial conditions (IC) profile data for the different CMAQ chemical mechanisms are provided with the software. After compiling the ICON source code in the $M3HOME/scripts/icon directory (see build instructions above), run the script "run.icon" to create a CMAQ-ready IC file. These data files will be written to the output directory defined in the ICON run script ($M3DATA/icon). The default run script will produce profile-based ICs for a 12 km CCTM simulation.
- Note: It's always a good idea to capture in a log file the text written to standard out when running these models. In each "run" script, near the top, is a suggested method (e.g., for ICON):
run.icon |& tee icon.log
Check the ICON log file to ensure complete and correct execution without errors.
Default boundary conditions (BC) profile data for the different CMAQ chemical mechanisms are also provided with the software. After compiling the BCON source code in the $M3HOME/scripts/bcon directory (see build instructions above), change directories to $M3HOME/scripts/bcon and run the following command to create a BC file for CMAQ:
run.bcon |& tee bcon.log
The default BCON run script also produces the profile-based BC dataset for a 12 km CCTM simulation. BCON may also be configured to produce boundary conditions for a nested simulation using output concentrations from the CCTM. See the CMAQv5x Operational Guidance Document for a discussion on creating inputs for nested simulations.
Next run the CCTM executable. Change directories to $M3HOME/scripts/cctm and run the command:
run.cctm |& tee cctm.log
By default, the CCTM runscript is set up for parallel processing. Users will likely need to configure the script for the processor configuration needed to submit parallel processing jobs on their systems.
Following completion of the CCTM run, you should have a complete collection of datasets which you can compare with the distribution datasets in DATA_REF.CMAQv5.1.Nov2015.tar.gz. Unless you modify the run scripts, the output data from all the models will be output to the following paths:
$M3DATA/ bcon/ cctm/ icon/
- You can check CMAQ output file headers (and data) using the netCDF utility ncdump.
- The GRIDDESC file written by MCIP contains horizontal projection and grid domain definitions that are required input for many CMAQ models. The CMAQ run scripts include an environment variable that points to the GRIDDESC file.
- The horizontal grid definition can be set to window from the met and emissions input files. However, the window must be a "proper subset" (i.e., a subset from the interior of the domain and not including boundaries). Note: The domains represented by the met and emissions data must be the same.
- Running CCTM for a windowed domain or a higher resolution nested domain from larger or coarser met and emissions datasets requires creating initial and boundary data for the target domain.
Tutorial/Benchmark Simulation Instructions
CMAQv5.1 is distributed with a complete set of input files required for running the CCTM. These data provide useful examples of the input files needed to run the CCTM. They can also be used to benchmark new installations of the software. Benchmarking refers to a simulation that is used to verify that the software is installed correctly. After running the CCTM with the tutorial data, compare the results of your simulation against the reference data to verify your results. Benchmarking is recommended in the following circumstances:
- Installation on a new server
- Following kernel upgrades
- Following Fortran compiler upgrades
- Following netCDF or I/O API library upgrades
To run the CMAQ tutorial simulation, change directories to the CCTM scripts directory and execute the CCTM run script:
cd $M3HOME/scripts/cctm run.cctm |& tee run.benchmark.log
Details of the CMAQv5.1 tutorial case:
- Grid: 12km CalNex study domain
- Period: 0Z July 1, 2011 - 0Z July 2, 2011
- Chemistry: EBI solver with the CB05e51 gas phase and AE6 aerosol mechanisms
- Inline windblown dust with BELD3 erodible agricultural land activity
- Inline lightning NO2
- Dynamic vertical diffusivity for urban/rural grid cells
- Inline deposition velocities
- Ammonia bi-directional flux for fertilizer application
- Surface HONO interaction
- Inline biogenic emissions
- Inline plume rise for large point sources
Notes on parallel (MPI) simulations
The CCTM is the only CMAQ program that can be run in parallel. The CCTM uses the Message Passing Interface (MPI) for managing calculations and communications across multiple processors. As each Linux server will have different characteristics, setting up your CMAQ jobs for MPI simulations will require server-specific configurations for compiling and running the CCTM on your server.
To compile the CCTM for an MPI simulation you should use the mpif90 wrapper script for your Fortran compiler. The config.cmaq script is already set up to use mpif90 in lieu of the standard compiler calls (i.e. ifort, pgf90, and gfortran).
The run script provided with the distribution shows an example for running the script on a system with intel_mpi, where NPROCS is set earlier in the script:
#> Executable call for multi PE, configure for your system set MPI = /usr/local/intel/impi/3.2.2.006/bin64 set MPIRUN = $MPI/mpirun time $MPIRUN -r ssh -np $NPROCS $BLD/$EXE
For a system using the LSF load-management software, the MPI wrapper may need to be called outside of the CCTM run script:
bsub -a intel_openmpi -n 16 mpirun.lsf run.cctm
Consult with your system administrator for instructions on how to submit MPI simulations on your cluster.
The CMAS Center conducted two sets of tests on the CMAQv5.1 release package. All tests used the EPA benchmark data that are distributed with the model. The compiler tests ran the default benchmark configuration with different compilers and MPI configurations. The configuration tests used the Portland Group 15.7 OpenMPI compiler to generate executables that exercise different scientific configurations of the release software.
|Scenario||Compiler||netCDF||I/O API||MPI_YN (#P)||MPI||Benchmark Timing
|Gfortran Serial||Gfortran version 4.8.1||4.1.1||3.1 (Nov 2015)||N||N/A||7:35:30|
|Gfortran MVAPICH2||Gfortran version 4.8.1||4.1.1||3.1 (Nov 2015)||Y (16)||mvapich2-1.7||0:42:40|
|Intel Serial||Intel Fortran version 16.0.0||4.1.1||3.1 (Nov 2015)||N||N/A||5:10:16|
|Intel OpenMPI||Intel Fortran version 16.0.0||4.1.1||3.1 (Nov 2015)||Y (16)||openMPI-1.4.2|
|Intel MVAPICH2||Intel Fortran version 16.0.0||4.1.1||3.1 (Nov 2015)||Y (16)||mvapich2-1.7|
|Portland Serial||PG Fortran version 15.7||4.1.1||3.1 (Nov 2015)||N||N/A||6:26:31|
|Portland OpenMPI||PG Fortran version 15.7||4.1.1||3.1 (Nov 2015)||Y (16)||openMPI-1.4.2||0:36:16|
|Portland MVAPICH2||PG Fortran version 15.7||4.1.1||3.1 (Nov 2015)||Y (16)||mvapich2-1.7||0:36:16|
|Benchmark Case||Online emissions processing, inline photolysis, lightning NOx, windblown dust, surface HONO, bidirectional NH3||cb05e51_ae6_aq|
|Multipollutant||Same as Benchmark case with MP mechanism enabled||cb05mp51_ae6_aq||Edits to MP species namelist files for benchmark case emissions species|
|Bidirectional Mercury||Same as Multipollutant with Hg BiDi activated||cb05mp51_ae6_aq||set CTM_HGBIDI = Y|
|Process Analysis||Benchmark case with IPR and IRR||ros3||Switch to Rosenbrock solver because EBI solver not supported by PA module|
|MOSAIC||Benchmark case with MOSAIC and additional stomatal flux files activated||cb05e51_ae6_aq|
|New Mechanism Test||Benchmark case with toluene and chlorine chemistry||cb05tucl_ae6_aq||Turned off NH3 Bidi and all diagnostic output files|
Shawn Roselle and Jeff Young, National Exposure Research Laboratory, U.S. EPA