CMAQv5.1 Readme file

From CMASWIKI
Jump to: navigation, search

Release Date: November 2015

Overview

This README file outlines the steps necessary to build and run the CMAQ version 5.1 (CMAQv5.1) release. The build and run scripts that are included in the tar files are set up to compile and run on Linux (see Testing Notes). To run CMAQv5.1 on another hardware platform, the C-shell scripts can be adapted for other Unix implementations.

The CMAQv5.1 release is available to the community accompanied by example scripts that invoke a specific configuration of the model. This configuration has been used by U.S. EPA and the CMAS Center in unit tests, system tests, and for model evaluation prior to the release of the model. There are other features and options within the CMAQ modeling system in this release that have not yet been fully tested, evaluated or documented. Feedback from the community on experiences using the CMAQ modeling system is welcome and can be provided via email to the contacts listed below.

The CMAQv5.1 distribution package includes:

  1. The base model and scripts package with the CMAQ source code, preprocessors, and utility programs
  2. Input data for a 12-km tutorial/benchmark CCTM simulation
  3. Reference output data for verifying the CMAQ installation

Base Model

  • CMAQv5.1.Nov2015.tar.gz - gzipped tar file containing source code for models, tools, and libraries, and C-Shell scripts to build and execute CMAQv5.1.

Instrumented Models

  • Anticipated release in 2016

NOTE: You must have I/O API version 3.1 and netCDF

Build Instructions

The following outlines a sequence of steps to build the base suite of CMAQ software, including the CMAQ Chemical Transport Model (CCTM).

Install the CMAQv5.1 source codes and scripts

In a working directory, gunzip and untar the model distribution tar file, CMAQv5.1.Nov2015.tar.gz. This will produce the following subdirectories: 

  CMAQv5.1/
            models/
            scripts/

Configure and source config.cmaq

Edit the file CMAQv5.1/scripts/config.cmaq to set the environment variable for M3HOME, and to set up your computer system information, including compilers and compiler flags. Environment variables M3MODEL and M3LIB are automatically set in the config.cmaq file. Navigate to the CMAQv5.1/scripts directory and use the following command to set the environment variables in the config file to set up the needed environment for building and running CMAQ. 

  cd CMAQv5.1/scripts
  source config.cmaq

Install test data

cd to $M3HOME and gunzip and untar the data tar file, DATA.CMAQv5.1.Nov2015.tar.gz. This will produce the following subdirectories: 

  data/
      bcon/
      bidi/
      crop/
      dust/
      emis/
      icon/
      lightning/
      mcip/
      ocean/
      phot/
      procan/

Install required external libraries

When you source the config.cmaq script the directory $M3HOME/lib ($M3LIB) will be created.

netCDF

The CMAQ build scripts require the netCDF library and include files in the $M3LIB path as $M3LIB/netcdf. If netCDF is installed elsewhere on your system, create a symbolic link in $M3LIB/netcdf to the existing netCDF. 

Example:

  cd $M3LIB
  ln -s /usr/local/lib/x86_64/ifc-13.1/netcdf-4.3.0 netcdf

I/O API

CMAQv5.1 requires I/O API version 3.1, download and install this library on your system following the instructions provided by the I/O API web site (http://cmascenter.org/ioapi/). The CMAQ build scripts look for the I/O API library in the $M3LIB path as $M3LIB/ioapi. If I/O API is installed elsewhere on your system, create symbolic links in $M3LIB/ioapi. 

Example to link for I/O API version 3.1:
  
  cd $M3LIB
  mkdir ioapi
  cd ioapi
  ln -s /usr/local/lib/x86_64/ifc-13.1/ioapi_3.1/Linux2_x86_64ifort lib
  ln -s /usr/local/lib/x86_64/ifc-13.1/ioapi_3.1/Linux2_x86_64ifort include
  ln -s /usr/local/lib/x86_64/ifc-13.1/ioapi_3.1/ioapi/fixed_src src

mpich

To run CMAQv5.1 across multiple processors, the "mpich" library and include files are required to build CMAQ. CMAQv5.1 has been successfully built and run with mvapich1, mvapich2, intel_mpi, and openmpi. The CMAQ build scripts look for the mpich library and include files in the $M3LIB path as $M3LIB/mpi. If mpich is installed elsewhere, create a symbolic link in $M3LIB/mpi. 

Example:

  cd $M3LIB
  ln -s /usr/local/lib/x86_64/ifc-13.1/mpich mpi

Build bldmake utility

Create bldmake, the tool required to build the executables for the CMAQ, model and supporting software and tools. 

  cd $M3HOME/scripts/build
  bldit.bldmake
Note: Utility bldmake will be installed under $M3LIB

Build CMAQ executables

Create the model executables: The initial conditions (ICs) preprocessor ICON and boundary conditions (BCs) preprocessor BCON need to be compiled and run separately to generate ICs/BCs from profile data or to generate ICs/BCs for a nested domain. Follow these steps to build the CMAQ executables: 

 cd $M3HOME/scripts/icon
 ./bldit.icon |& tee bldit.icon.profile.log
 cd $M3HOME/scripts/bcon
 ./bldit.bcon |& tee bldit.bcon.profile.log
 cd $M3HOME/scripts/cctm
 ./bldit.cctm |& tee bldit.cctm.log

The meteorology preprocessor MCIP uses a Makefile rather than bldit scripts. Navigate to the MCIP source code directory and configure the Makefile for your Fortran compiler. Run the command "make" to build the MCIP executable 

 cd $M3HOME/scripts/mcip/src
 [edit and save the Makefile]
 make |& tee make.mcip.log

Run Instructions

The CMAQv5.1 distribution package includes a full set of test input data needed to run the CCTM for one day. Instructions are provided below for the general approach for running CMAQ and for running the CMAQ tutorial case, which only requires running the CCTM.

General Simulation Instructions

These steps assume that the user has available WRF meteorology and SMOKE emissions data.

  1. Run MCIP to prepare the CMAQ-ready meteorological data from the WRF output
  2. Run ICON and BCON to create initial and boundary conditions input data
  3. Run the CCTM to estimate air quality fields

Although MCIP meteorology data are provided with the CMAQv5.1 test data, MCIP4.3 code and scripts are packaged in the CMAQv5.1 distribution. Users can compile and run MCIP to process WRF meteorology data for input to CMAQ and SMOKE.

ICON

Default initial conditions (IC) profile data for the different CMAQ chemical mechanisms are provided with the software. After compiling the ICON source code in the $M3HOME/scripts/icon directory (see build instructions above), run the script "run.icon" to create a CMAQ-ready IC file. These data files will be written to the output directory defined in the ICON run script ($M3DATA/icon). The default run script will produce profile-based ICs for a 12 km CCTM simulation.

Note: It's always a good idea to capture in a log file the text written to standard out when running these models. In each "run" script, near the top, is a suggested method (e.g., for ICON):
   run.icon |& tee icon.log

Check the ICON log file to ensure complete and correct execution without errors.

BCON

Default boundary conditions (BC) profile data for the different CMAQ chemical mechanisms are also provided with the software. After compiling the BCON source code in the $M3HOME/scripts/bcon directory (see build instructions above), change directories to $M3HOME/scripts/bcon and run the following command to create a BC file for CMAQ: 

run.bcon |& tee bcon.log

The default BCON run script also produces the profile-based BC dataset for a 12 km CCTM simulation. BCON may also be configured to produce boundary conditions for a nested simulation using output concentrations from the CCTM. See the CMAQv5x Operational Guidance Document for a discussion on creating inputs for nested simulations.

CCTM

Next run the CCTM executable. Change directories to $M3HOME/scripts/cctm and run the command: 

run.cctm |& tee cctm.log

By default, the CCTM runscript is set up for parallel processing. Users will likely need to configure the script for the processor configuration needed to submit parallel processing jobs on their systems.

Following completion of the CCTM run, you should have a complete collection of datasets which you can compare with the distribution datasets in DATA_REF.CMAQv5.1.Nov2015.tar.gz. Unless you modify the run scripts, the output data from all the models will be output to the following paths: 

   $M3DATA/
          bcon/
          cctm/
          icon/

Other details

  • You can check CMAQ output file headers (and data) using the netCDF utility ncdump.
  • The GRIDDESC file written by MCIP contains horizontal projection and grid domain definitions that are required input for many CMAQ models. The CMAQ run scripts include an environment variable that points to the GRIDDESC file.
  • The horizontal grid definition can be set to window from the met and emissions input files. However, the window must be a "proper subset" (i.e., a subset from the interior of the domain and not including boundaries). Note: The domains represented by the met and emissions data must be the same.
  • Running CCTM for a windowed domain or a higher resolution nested domain from larger or coarser met and emissions datasets requires creating initial and boundary data for the target domain.

Tutorial/Benchmark Simulation Instructions

CMAQv5.1 is distributed with a complete set of input files required for running the CCTM. These data provide useful examples of the input files needed to run the CCTM. They can also be used to benchmark new installations of the software. Benchmarking refers to a simulation that is used to verify that the software is installed correctly. After running the CCTM with the tutorial data, compare the results of your simulation against the reference data to verify your results. Benchmarking is recommended in the following circumstances:

  • Installation on a new server
  • Following kernel upgrades
  • Following Fortran compiler upgrades
  • Following netCDF or I/O API library upgrades

To run the CMAQ tutorial simulation, change directories to the CCTM scripts directory and execute the CCTM run script: 

 cd $M3HOME/scripts/cctm
 run.cctm |& tee run.benchmark.log

Details of the CMAQv5.1 tutorial case:

  • Grid: 12km CalNex study domain
  • Period: 0Z July 1, 2011 - 0Z July 2, 2011
  • Chemistry: EBI solver with the CB05e51 gas phase and AE6 aerosol mechanisms
  • Inline windblown dust with BELD3 erodible agricultural land activity
  • Inline lightning NO2
  • Dynamic vertical diffusivity for urban/rural grid cells
  • Inline deposition velocities
  • Ammonia bi-directional flux for fertilizer application
  • Surface HONO interaction
  • Inline biogenic emissions
  • Inline plume rise for large point sources

Notes on parallel (MPI) simulations

The CCTM is the only CMAQ program that can be run in parallel. The CCTM uses the Message Passing Interface (MPI) for managing calculations and communications across multiple processors. As each Linux server will have different characteristics, setting up your CMAQ jobs for MPI simulations will require server-specific configurations for compiling and running the CCTM on your server.

To compile the CCTM for an MPI simulation you should use the mpif90 wrapper script for your Fortran compiler. The config.cmaq script is already set up to use mpif90 in lieu of the standard compiler calls (i.e. ifort, pgf90, and gfortran).

The run script provided with the distribution shows an example for running the script on a system with intel_mpi, where NPROCS is set earlier in the script: 

 #> Executable call for multi PE, configure for your system 
  set MPI = /usr/local/intel/impi/3.2.2.006/bin64
  set MPIRUN = $MPI/mpirun
  time $MPIRUN -r ssh -np $NPROCS $BLD/$EXE

For a system using the LSF load-management software, the MPI wrapper may need to be called outside of the CCTM run script: 

 bsub -a intel_openmpi -n 16 mpirun.lsf run.cctm

Consult with your system administrator for instructions on how to submit MPI simulations on your cluster.

Testing Procedures

The CMAS Center conducted two sets of tests on the CMAQv5.1 release package. All tests used the EPA benchmark data that are distributed with the model. The compiler tests ran the default benchmark configuration with different compilers and MPI configurations. The configuration tests used the Portland Group 15.7 OpenMPI compiler to generate executables that exercise different scientific configurations of the release software.


Table 1. CMAQv5.1 compilation testing manifest
Scenario Compiler netCDF I/O API MPI_YN (#P) MPI Benchmark Timing
(HH:MM:SS)		 Notes
Gfortran Serial Gfortran version 4.8.1 4.1.1 3.1 (Nov 2015) N N/A 7:35:30
Gfortran MVAPICH2 Gfortran version 4.8.1 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 0:42:40
Intel Serial Intel Fortran version 16.0.0 4.1.1 3.1 (Nov 2015) N N/A 5:10:16
Intel OpenMPI Intel Fortran version 16.0.0 4.1.1 3.1 (Nov 2015) Y (16) openMPI-1.4.2
Intel MVAPICH2 Intel Fortran version 16.0.0 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7
Portland Serial PG Fortran version 15.7 4.1.1 3.1 (Nov 2015) N N/A 6:26:31
Portland OpenMPI PG Fortran version 15.7 4.1.1 3.1 (Nov 2015) Y (16) openMPI-1.4.2 0:36:16
Portland MVAPICH2 PG Fortran version 15.7 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 0:36:16
Table 2. CMAQv5.1 configuration testing manifest
Scenario Description Mechanism Notes
Benchmark Case Online emissions processing, inline photolysis, lightning NOx, windblown dust, surface HONO, bidirectional NH3 cb05e51_ae6_aq
Multipollutant Same as Benchmark case with MP mechanism enabled cb05mp51_ae6_aq Edits to MP species namelist files for benchmark case emissions species
Bidirectional Mercury Same as Multipollutant with Hg BiDi activated cb05mp51_ae6_aq set CTM_HGBIDI = Y
Process Analysis Benchmark case with IPR and IRR ros3 Switch to Rosenbrock solver because EBI solver not supported by PA module
MOSAIC Benchmark case with MOSAIC and additional stomatal flux files activated cb05e51_ae6_aq
New Mechanism Test Benchmark case with toluene and chlorine chemistry cb05tucl_ae6_aq Turned off NH3 Bidi and all diagnostic output files

References

Contact

Shawn Roselle and Jeff Young, National Exposure Research Laboratory, U.S. EPA

Retrieved from "https://www.airqualitymodeling.org/index.php?title=CMAQv5.1_Readme_file&oldid=794"