CMAQ version 5.2 DDM Technical Documentation

From CMASWIKI
Jump to: navigation, search

Release Notes for CMAQv5.2_DDM - Target: September 2017

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.2 DDM is being tested prior to release

The CMAQv5.2 DDM official release repository will be available from the U.S. EPA GitHub site.

Markdown draft version of the documentation for CMAQv5.2 DDM, including build instructions and technical features is currently available from [1].

Release Testing

The CMAS Center will test the CMAQv5.2 DDM release package with the Portland Group, Intel, and GNU Fortran compilers. The CMAQv5.2 Technical Notes contains timing information for the base release. The results of the CMAS Center tests for the CMAQv5.2 DDM release, including run times for different compilers and configurations will be provided below. The CMAQv5.2 DDM tests were conducted with the U.S. EPA benchmark domain testing dataset for a single day: July 1, 2011.

DDM changes to bldit_cctm.csh:

set Mechanism = cb6r3_ae6nvPOA_aq
set Ddm3d     = ddm3d
set sens      = ( -Dsens )

DDM changes to run_cctm.csh:

set MECH      = cb6r3_ae6nvPOA_aq
setenv CTM_ABFLUX N
setenv CTM_GRAV_SETL N

sensinput.cmas.dat contains:

ENX
 EMIS
  GRID
  1
  EMIS_1
 SPECIES
  NO2
  NO
ESX
 EMIS
  GRID
  1
  EMIS_1
 SPECIES
  SO2
END


See example run script for all of the settings under this heading:

  • # ===================================================================
  • #> CMAQ-DDM-3D
  • # ===================================================================
set NPMAX    = 2
setenv SEN_INPUT <path>/sensinput.cmas.dat
setenv DDM3D_HIGH N   # allow higher-order sensitivity parameters
setenv DDM3D_RGN N    # allows for a file specifying regions
setenv REGIONS_1
setenv DDM3D_BCRGN N  # allows for a file specifying boundary regions
setenv BCREGIONS_1
setenv DDM3D_ES Y     # emissions split into categories  
setenv DDM3D_BCS F      # use sensitivity bc file for nested runs                                                                                                                                                           

Compiler flags:

PGI: -Mfixed -O3 -Mextend
GCC: -ffixed-form -ffixed-line-length-132 -O3 -funroll-loops -finit-character=32
Intel: -fixed -132 -O3 -override-limits -fno-alias -mp1 -fp-model precise -fp-model source -shared-intel -openmp
In the Intel Basic Test: -fixed -132 -O3 -openmp
In the NoOpt Tests: -O0 with extend source and fixed line length flags

Scatter plots

Concentration at the location of the domain-wide max of the absolute value of the difference for each timestep between the sensitivity cases and the base case for CMAQ v5.2 DDM which is using mvapich and the intel compiler version 15.0 (mvapich2_intel15.0).

Sensitivity cases include:

Github release version of the CMAQ v5.2 code with matching MECH and flags to the DDM case on 16 pe.

CCTM_v52_gcc_SE52BENCH_3

CMAQ v5.2 code + DDM code, and matching MECH and flags to the DDM case, but with the compile done with the noDDM option on 16 pe.

CCTM_v52_gcc_SE52BENCH_ddm_noDDM

CMAQ v5.2 code + DDM code, compiled with DDM option turned on

mvapich2_intel15.0_base (16 pe)
mvapich2_gcc4.8.1_16pe
mvapich2_gcc4.8.1_32pe
openmpi_gcc_4.8.1_16pe
mvapich2_pgi15.7_16pe
serial_mvapich2_pgi15.7_1pe

Google Drive location of DDM Testing Plots

UNC module pgi/16.1
Table 1. CMAQv5.2 DDM compilation testing manifest
Scenario Compiler netCDF I/O API MPI_YN (#P) MPI CMAQv5.2 Timing
(HH:MM:SS)
CMAQv5.2 DDM Timing
(HH:MM:SS)
Notes
Gfortran openmpi Gfortran version 4.8.1 4.1.1 3.1 (Nov 2015) Y (16) openmpi 34:13.15 1:45:33 UNC module openmpi_gcc/4.8.1
Gfortran MVAPICH2 Gfortran version 4.8.1 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 1:07:23 UNC module mvapich2_gcc/4.8.1
Intel MVAPICH2 Intel Fortran version 15.0 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 1:33:10 UNC module mvapich2_intel/15.0
Portland Serial PG Fortran version 16.1 4.1.1 3.1 (Nov 2015) N N/A 31:24:04 UNC module pgi/16.1
Portland MVAPICH2 PGI Fortran version 15.7 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 2:11:40 UNC module mvapich2_pgi/15.7
Portland MVAPICH2 PGI Fortran version 15.7 4.1.1 3.1 (Nov 2015) Y (32) mvapich2-1.7 1:07:22 UNC module mvapich2_pgi/15.7