CMAQ version 5.3beta (November 2018 release) Technical Documentation

Release Notes for CMAQv5.3beta - November 2018

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3 (CMAQv5.3) is scheduled for release in early 2019.
To facilitate community contributions to this model release, a “Beta” version was created and made available to external collaborators/developers in November 2018. This Beta-version will undergo extensive application and evaluation over the next few months, which may lead to additional changes before the final release.

The CMAQv5.3beta repository is available from the U.S. EPA GitHub CMAQ repository.

Complete documentation for CMAQv5.3beta, including build instructions and technical features are available from the CMAQv5.3beta GitHub page.

Release Testing

The CMAS Center will test the CMAQv5.3beta release package with several versions and configurations of the Portland Group, Intel, and GNU Fortran compilers. In addition to different serial and parallel configurations, the release was tested under different science configurations. The results of the CMAS Center tests will include run times for different compilers and configurations are provided below. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. This dataset is distributed with CMAQv5.3beta to use for benchmarking the model installation and includes new EPIC files.

Two classes of tests:

• Compiler tests used the default benchmark configuration with different compilers and MPI configurations.
• Configuration tests used the gcc or intel vXX OpenMPI compiler to generate executables that exercise different scientific configurations of the release software.

Compiler flags:

• GCC: mpif90 -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising
• Intel: mpif90 -c -fixed -132 -O3 -fno-alias -mp1 -fp-model source
• In the Intel Basic Test: -fixed -132 -O3 -openmp
• In the NoOpt Tests: -O0 with extend source and fixed line length flags

Note: this website contains explanation of why timing varies depending on the mpi flavor that is being used, and suggests flags to pin or bind tasks to a core set of cpus,and also options to use the infiniband. https://www.chpc.utah.edu/documentation/software/mpilibraries.php http://www.nersc.gov/users/computational-systems/cori/running-jobs/example-batch-scripts/

Specs for Dogwood machine: https://help.unc.edu/files/2018/03/dogwood.cheatsheet.pdf

R script for comparing model output results from different compilers

• currently running the script on killdevil, which has an older version of R and the packages required
• module load r/2.15.1
• setenv R_LIBS /nas01/depts/ie/cempd/apps/AMET/AMET2/local/R_2.15.1
• Dongmei is building the packages on longleaf, for us to be able to run the script there.

| Compilation Testing Plots

Table 1. CMAQv5.3beta compilation testing manifest

Scenario	Compiler	netCDF	I/O API	MPI_YN (#P)	MPI	CMAQv5.3.beta Timing (HH:MM:SS)	Notes
Gfortran Serial (killdevil)	Gfortran version 6.3.0	v4.1.3	3.2	N	N/A		UNC module gcc/4.8.1
Gfortran MVAPICH2	Gfortran version 4.8.1	v4.1.3	3.2	Y (64)	mvapich2-1.7
Gfortran MPICH (dogwood)	Gfortran version 4.8.1	4.6.0	3.2	Y (64)	mpich/3.2.1_1	558.7	Linuxbrew gcc
Intel Serial (killdevil)	Intel Fortran version 16.2.0	4.3.2	3.1 (Nov 2015)	N	N/A		UNC module intel/16.2
Intel OpenMPI (EPA Config)	Intel Fortran version 15.0.0	4.1.3	3.2	Y (64)	openMPI-		EPA module
Intel OpenMPI (dogwood)	Intel Fortran version 17.2	4.1.3	3.2	Y (64)	openmpi-v3.0.2	616.5	UNC module openmpi_3.0.0/intel_17.2
Intel MVAPICH2 (dogwood)	Intel Fortran version 17.2	4.1.3	3.2	Y (64)	mvapich2_2.3rc1	399.2 (Mark Reed)	mvapich2_2.3rc1/intel_17.2 timing log available: https://github.com/USEPA/CMAQ/issues/118
Intel mvapich2 (killdevil)	Intel Fortran version 15.0	4.3.2	3.1 (Nov 2015)	Y (32)	openMPI-1.4.2	1118.04	UNC module openmpi_intel/15.0; basic compiler flags for comparison to PGI and GCC
Intel mvapich2 (killdevil)	Intel Fortran version 15.0	4.3.2	3.1 (Nov 2015)	Y (64)	openMPI-1.4.2	886.0	UNC module openmpi_intel/15.0; basic compiler flags for comparison to PGI and GCC
Portland Serial (killdevil)	PG Fortran version 16.1	4.3.2	3.1 (Nov 2015)	N	N/A		Killdevil UNC module
Gfortran OpenMPI (NoOpt)	Gfortran version 4.8.1	4.3.2	3.1 (Nov 2015)	Y (16)	openMPI-1.4.2		UNC module openmpi_gcc/4.8.1; optimization -O0

Table 2. CMAQv5.3beta configuration testing manifest

Scenario	Description	Mechanism	Notes	Timing (32 PE)
Benchmark Case	Online emissions processing, inline photolysis, inline lightning from MCIP RC, no windblown dust, surface HONO, bidirectional NH3 and Hg, no potential vorticity scaling	cb05e51_ae6_aq	Pending; LTNGNO InLine, LTNGPARM = N, LOG_START = 2.0	0:40:20
Halogen Chemistry	Same as Benchmark case with halogen chemistry enabled	cb05eh51_ae6_aq	Pending. Turned off the diagnostic file logging.
No Bidi	Same as Benchmark with Hg and NH3 BiDi deactivated	cb05e51_ae6_aq	Pending. set CTM_HGBIDI = N; set ABFLUX = N
Process Analysis	Benchmark case with IPR and IRR	ros3	Pending. Switch to Rosenbrock solver because EBI solver not supported by PA module; ran with inline process analysis
MOSAIC	Benchmark case with MOSAIC and additional stomatal flux files activated	cb05e51_ae6_aq	Pending. set CTM_MOSAIC = Y; set CTM_FST = Y
New Mechanism Test	Benchmark case with toluene and chlorine chemistry	cb05tucl_ae6_aq	Pending.
Potential vorticity UTLS exchange	Benchmark case with scaling for STE	cb05e51_ae6_aq	Pending: Uncomment potvortO3 in build. Need PV variable in METCRO3D file
Dust	Benchmark case with dust, including new MODIS FP input	cb05e51_ae6_aq	Pending. setenv CTM_WB_DUST Y; setenv CTM_ERODE_AGLAND Y; setenv CTM_WBDUST_BELD BELD3
Hourly NLDN	Benchmark with lightning NOx calculated using hourly bNLDN strikes	cb05e51_ae6_aq	Pending; LTNGNO InLine, LTNGPARM = Y, USE_NLDN Y
POA Sensitivity	Benchmark with new POA mechanism	cb05e51_ae6nvPOA_aq	Pending;