View source for CMAQ version 5.2 DDM Technical Documentation

= Release Notes for CMAQv5.2_DDM - Target: September 2017  =

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.2 DDM is being tested prior to release

The CMAQv5.2 DDM official release repository will be available from the U.S. EPA GitHub site. 

Markdown draft version of the documentation for CMAQv5.2 DDM, including build instructions and technical features is currently available from [https://github.com/lizadams/CMAQ/tree/5.2/DOCS/Instrumented_Docs].

= Release Testing =

The CMAS Center will test the CMAQv5.2 DDM release package with the Portland Group, Intel, and GNU Fortran compilers.  The [https://www.airqualitymodeling.org/index.php/CMAQ_version_5.2_(June_2017_release)_Technical_Documentation CMAQv5.2 Technical Notes] contains timing information for the base release. The results of the CMAS Center tests for the CMAQv5.2 DDM release, including run times for different compilers and configurations will be provided below. The CMAQv5.2 DDM tests were conducted with the U.S. EPA benchmark domain testing dataset for a single day: July 1, 2011.  

DDM changes to bldit_cctm.csh:
 set Mechanism = cb6r3_ae6nvPOA_aq
 set Ddm3d     = ddm3d
 set sens      = ( -Dsens )

DDM changes to run_cctm.csh:
 set MECH      = cb6r3_ae6nvPOA_aq
 setenv CTM_ABFLUX N
 setenv CTM_GRAV_SETL N

sensinput.cmas.dat contains:

 ENX
  EMIS
   GRID
   1
   EMIS_1
  SPECIES
   NO2
   NO
 ESX
  EMIS
   GRID
   1
   EMIS_1
  SPECIES
   SO2
 END


See example run script for all of the settings under this heading:
* # ===================================================================
* #> CMAQ-DDM-3D
* # ===================================================================  
  
 set NPMAX    = 2
 setenv SEN_INPUT <path>/sensinput.cmas.dat
 setenv DDM3D_HIGH N   # allow higher-order sensitivity parameters
 setenv DDM3D_RGN N    # allows for a file specifying regions
 setenv REGIONS_1
 setenv DDM3D_BCRGN N  # allows for a file specifying boundary regions
 setenv BCREGIONS_1
 setenv DDM3D_ES Y     # emissions split into categories  
 setenv DDM3D_BCS F      # use sensitivity bc file for nested runs                                                                                                                                                           

Compiler flags:
 PGI: -Mfixed -O3 -Mextend
 GCC: -ffixed-form -ffixed-line-length-132 -O3 -funroll-loops -finit-character=32
 Intel: -fixed -132 -O3 -override-limits -fno-alias -mp1 -fp-model precise -fp-model source -shared-intel -openmp
 In the Intel Basic Test: -fixed -132 -O3 -openmp
 In the NoOpt Tests: -O0 with extend source and fixed line length flags

== Scatter plots == 
Concentration at the location of the domain-wide max of the absolute value of the difference for each timestep
between the sensitivity cases and the base case for CMAQ v5.2 DDM which is using mvapich and the intel compiler version 15.0 (mvapich2_intel15.0).

Sensitivity cases include:

Github release version of the CMAQ v5.2 code with matching MECH and flags to the DDM case on 16 pe.
 CCTM_v52_gcc_SE52BENCH_3

CMAQ v5.2 code + DDM code, and matching MECH and flags to the DDM case, but with the compile done with the noDDM option on 16 pe.
 CCTM_v52_gcc_SE52BENCH_ddm_noDDM

CMAQ v5.2 code + DDM code, compiled with DDM option turned on
 mvapich2_intel15.0_base (16 pe)
 mvapich2_gcc4.8.1_16pe
 mvapich2_gcc4.8.1_32pe
 openmpi_gcc_4.8.1_16pe
 mvapich2_pgi15.7_16pe
 serial_mvapich2_pgi15.7_1pe

[https://drive.google.com/drive/folders/0B3Xmjif6GtQZUy1pSHZpc1dBLXM Google Drive location of DDM Testing Plots]

{| class="wikitable"
|+ Table 1. CMAQv5.2 DDM compilation testing manifest
|-
!Scenario !! Compiler !! netCDF !! I/O API !! MPI_YN (#P) !! MPI !! CMAQv5.2 Timing<br>(HH:MM:SS) !! CMAQv5.2 DDM Timing<br>(HH:MM:SS) !! Notes
|-
|Gfortran openmpi
|Gfortran version 4.8.1
|4.1.1
|3.1 (Nov 2015)
|Y (16)
|openmpi
|34:13.15
|1:45:33
|UNC module openmpi_gcc/4.8.1
|-
|Gfortran MVAPICH2
|Gfortran version 4.8.1
|4.1.1
|3.1 (Nov 2015)
|Y (16)
|mvapich2-1.7
|
|1:07:23
|UNC module mvapich2_gcc/4.8.1
|-
|Intel MVAPICH2
|Intel Fortran version 15.0
|4.1.1
|3.1 (Nov 2015)
|Y (16)
|mvapich2-1.7
|
|1:33:10
|UNC module mvapich2_intel/15.0
|-
|Portland Serial
|PG Fortran version 16.1
|4.1.1
|3.1 (Nov 2015)
|N
|N/A
|
|31:24:04
|UNC module pgi/16.1
|-
|Portland MVAPICH2
|PGI Fortran version 15.7
|4.1.1
|3.1 (Nov 2015)
|Y (16)
|mvapich2-1.7
|
|2:11:40
|UNC module mvapich2_pgi/15.7
|-
UNC module pgi/16.1
|-
|Portland MVAPICH2
|PGI Fortran version 15.7
|4.1.1
|3.1 (Nov 2015)
|Y (32)
|mvapich2-1.7
|
|1:07:22
|UNC module mvapich2_pgi/15.7
|}