Difference between revisions of "CMAQ version 5.2 DDM Technical Documentation"

From CMASWIKI
Jump to: navigation, search
(Release Testing)
(Release Testing)
 
(18 intermediate revisions by the same user not shown)
Line 10: Line 10:
  
 
The CMAS Center will test the CMAQv5.2 DDM release package with the Portland Group, Intel, and GNU Fortran compilers.  The [https://www.airqualitymodeling.org/index.php/CMAQ_version_5.2_(June_2017_release)_Technical_Documentation CMAQv5.2 Technical Notes] contains timing information for the base release. The results of the CMAS Center tests for the CMAQv5.2 DDM release, including run times for different compilers and configurations will be provided below. The CMAQv5.2 DDM tests were conducted with the U.S. EPA benchmark domain testing dataset for a single day: July 1, 2011.   
 
The CMAS Center will test the CMAQv5.2 DDM release package with the Portland Group, Intel, and GNU Fortran compilers.  The [https://www.airqualitymodeling.org/index.php/CMAQ_version_5.2_(June_2017_release)_Technical_Documentation CMAQv5.2 Technical Notes] contains timing information for the base release. The results of the CMAS Center tests for the CMAQv5.2 DDM release, including run times for different compilers and configurations will be provided below. The CMAQv5.2 DDM tests were conducted with the U.S. EPA benchmark domain testing dataset for a single day: July 1, 2011.   
 +
 +
DDM changes to bldit_cctm.csh:
 +
set Mechanism = cb6r3_ae6nvPOA_aq
 +
set Ddm3d    = ddm3d
 +
set sens      = ( -Dsens )
 +
 +
DDM changes to run_cctm.csh:
 +
set MECH      = cb6r3_ae6nvPOA_aq
 +
setenv CTM_ABFLUX N
 +
setenv CTM_GRAV_SETL N
 +
 +
sensinput.cmas.dat contains:
 +
 +
ENX
 +
  EMIS
 +
  GRID
 +
  1
 +
  EMIS_1
 +
  SPECIES
 +
  NO2
 +
  NO
 +
ESX
 +
  EMIS
 +
  GRID
 +
  1
 +
  EMIS_1
 +
  SPECIES
 +
  SO2
 +
END
 +
 +
 +
See example run script for all of the settings under this heading:
 +
* # ===================================================================
 +
* #> CMAQ-DDM-3D
 +
* # =================================================================== 
 +
 
 +
set NPMAX    = 2
 +
setenv SEN_INPUT <path>/sensinput.cmas.dat
 +
setenv DDM3D_HIGH N  # allow higher-order sensitivity parameters
 +
setenv DDM3D_RGN N    # allows for a file specifying regions
 +
setenv REGIONS_1
 +
setenv DDM3D_BCRGN N  # allows for a file specifying boundary regions
 +
setenv BCREGIONS_1
 +
setenv DDM3D_ES Y    # emissions split into categories 
 +
setenv DDM3D_BCS F      # use sensitivity bc file for nested runs                                                                                                                                                         
  
 
Compiler flags:
 
Compiler flags:
* PGI: -Mfixed -O3 -Mextend
+
PGI: -Mfixed -O3 -Mextend
* GCC: -ffixed-form -ffixed-line-length-132 -O3 -funroll-loops -finit-character=32
+
GCC: -ffixed-form -ffixed-line-length-132 -O3 -funroll-loops -finit-character=32
* Intel: -fixed -132 -O3 -override-limits -fno-alias -mp1 -fp-model precise -fp-model source -shared-intel -openmp
+
Intel: -fixed -132 -O3 -override-limits -fno-alias -mp1 -fp-model precise -fp-model source -shared-intel -openmp
* In the Intel Basic Test: -fixed -132 -O3 -openmp
+
In the Intel Basic Test: -fixed -132 -O3 -openmp
* In the NoOpt Tests: -O0 with extend source and fixed line length flags
+
In the NoOpt Tests: -O0 with extend source and fixed line length flags
  
[Google Drive location of Compilation Testing Plots]
+
== Scatter plots ==
 +
Concentration at the location of the domain-wide max of the absolute value of the difference for each timestep
 +
between the sensitivity cases and the base case for CMAQ v5.2 DDM which is using mvapich and the intel compiler version 15.0 (mvapich2_intel15.0).
 +
 
 +
Sensitivity cases include:
 +
 
 +
Github release version of the CMAQ v5.2 code with matching MECH and flags to the DDM case on 16 pe.
 +
CCTM_v52_gcc_SE52BENCH_3
 +
 
 +
CMAQ v5.2 code + DDM code, and matching MECH and flags to the DDM case, but with the compile done with the noDDM option on 16 pe.
 +
CCTM_v52_gcc_SE52BENCH_ddm_noDDM
 +
 
 +
CMAQ v5.2 code + DDM code, compiled with DDM option turned on
 +
mvapich2_intel15.0_base (16 pe)
 +
mvapich2_gcc4.8.1_16pe
 +
mvapich2_gcc4.8.1_32pe
 +
openmpi_gcc_4.8.1_16pe
 +
mvapich2_pgi15.7_16pe
 +
serial_mvapich2_pgi15.7_1pe
 +
 
 +
[https://drive.google.com/drive/folders/0B3Xmjif6GtQZUy1pSHZpc1dBLXM Google Drive location of DDM Testing Plots]
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 31: Line 96:
 
|Y (16)
 
|Y (16)
 
|openmpi
 
|openmpi
|
+
|34:13.15
|
+
|1:45:33
 
|UNC module openmpi_gcc/4.8.1
 
|UNC module openmpi_gcc/4.8.1
 
|-
 
|-
Line 42: Line 107:
 
|mvapich2-1.7
 
|mvapich2-1.7
 
|
 
|
|
+
|1:07:23
 
|UNC module mvapich2_gcc/4.8.1
 
|UNC module mvapich2_gcc/4.8.1
 
|-
 
|-
Line 52: Line 117:
 
|mvapich2-1.7
 
|mvapich2-1.7
 
|
 
|
|
+
|1:33:10
 
|UNC module mvapich2_intel/15.0
 
|UNC module mvapich2_intel/15.0
 
|-
 
|-
Line 62: Line 127:
 
|N/A
 
|N/A
 
|
 
|
|
+
|31:24:04
 
|UNC module pgi/16.1
 
|UNC module pgi/16.1
 
|-
 
|-
Line 71: Line 136:
 
|Y (16)
 
|Y (16)
 
|mvapich2-1.7
 
|mvapich2-1.7
|34:13.15
+
|
|1:43:47
+
|2:11:40
 
|UNC module mvapich2_pgi/15.7
 
|UNC module mvapich2_pgi/15.7
 
|-
 
|-

Latest revision as of 15:30, 20 September 2017

Release Notes for CMAQv5.2_DDM - Target: September 2017

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.2 DDM is being tested prior to release

The CMAQv5.2 DDM official release repository will be available from the U.S. EPA GitHub site.

Markdown draft version of the documentation for CMAQv5.2 DDM, including build instructions and technical features is currently available from [1].

Release Testing

The CMAS Center will test the CMAQv5.2 DDM release package with the Portland Group, Intel, and GNU Fortran compilers. The CMAQv5.2 Technical Notes contains timing information for the base release. The results of the CMAS Center tests for the CMAQv5.2 DDM release, including run times for different compilers and configurations will be provided below. The CMAQv5.2 DDM tests were conducted with the U.S. EPA benchmark domain testing dataset for a single day: July 1, 2011.

DDM changes to bldit_cctm.csh:

set Mechanism = cb6r3_ae6nvPOA_aq
set Ddm3d     = ddm3d
set sens      = ( -Dsens )

DDM changes to run_cctm.csh:

set MECH      = cb6r3_ae6nvPOA_aq
setenv CTM_ABFLUX N
setenv CTM_GRAV_SETL N

sensinput.cmas.dat contains:

ENX
 EMIS
  GRID
  1
  EMIS_1
 SPECIES
  NO2
  NO
ESX
 EMIS
  GRID
  1
  EMIS_1
 SPECIES
  SO2
END


See example run script for all of the settings under this heading:

  • # ===================================================================
  • #> CMAQ-DDM-3D
  • # ===================================================================
set NPMAX    = 2
setenv SEN_INPUT <path>/sensinput.cmas.dat
setenv DDM3D_HIGH N   # allow higher-order sensitivity parameters
setenv DDM3D_RGN N    # allows for a file specifying regions
setenv REGIONS_1
setenv DDM3D_BCRGN N  # allows for a file specifying boundary regions
setenv BCREGIONS_1
setenv DDM3D_ES Y     # emissions split into categories  
setenv DDM3D_BCS F      # use sensitivity bc file for nested runs                                                                                                                                                           

Compiler flags:

PGI: -Mfixed -O3 -Mextend
GCC: -ffixed-form -ffixed-line-length-132 -O3 -funroll-loops -finit-character=32
Intel: -fixed -132 -O3 -override-limits -fno-alias -mp1 -fp-model precise -fp-model source -shared-intel -openmp
In the Intel Basic Test: -fixed -132 -O3 -openmp
In the NoOpt Tests: -O0 with extend source and fixed line length flags

Scatter plots

Concentration at the location of the domain-wide max of the absolute value of the difference for each timestep between the sensitivity cases and the base case for CMAQ v5.2 DDM which is using mvapich and the intel compiler version 15.0 (mvapich2_intel15.0).

Sensitivity cases include:

Github release version of the CMAQ v5.2 code with matching MECH and flags to the DDM case on 16 pe.

CCTM_v52_gcc_SE52BENCH_3

CMAQ v5.2 code + DDM code, and matching MECH and flags to the DDM case, but with the compile done with the noDDM option on 16 pe.

CCTM_v52_gcc_SE52BENCH_ddm_noDDM

CMAQ v5.2 code + DDM code, compiled with DDM option turned on

mvapich2_intel15.0_base (16 pe)
mvapich2_gcc4.8.1_16pe
mvapich2_gcc4.8.1_32pe
openmpi_gcc_4.8.1_16pe
mvapich2_pgi15.7_16pe
serial_mvapich2_pgi15.7_1pe

Google Drive location of DDM Testing Plots

UNC module pgi/16.1
Table 1. CMAQv5.2 DDM compilation testing manifest
Scenario Compiler netCDF I/O API MPI_YN (#P) MPI CMAQv5.2 Timing
(HH:MM:SS)
CMAQv5.2 DDM Timing
(HH:MM:SS)
Notes
Gfortran openmpi Gfortran version 4.8.1 4.1.1 3.1 (Nov 2015) Y (16) openmpi 34:13.15 1:45:33 UNC module openmpi_gcc/4.8.1
Gfortran MVAPICH2 Gfortran version 4.8.1 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 1:07:23 UNC module mvapich2_gcc/4.8.1
Intel MVAPICH2 Intel Fortran version 15.0 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 1:33:10 UNC module mvapich2_intel/15.0
Portland Serial PG Fortran version 16.1 4.1.1 3.1 (Nov 2015) N N/A 31:24:04 UNC module pgi/16.1
Portland MVAPICH2 PGI Fortran version 15.7 4.1.1 3.1 (Nov 2015) Y (16) mvapich2-1.7 2:11:40 UNC module mvapich2_pgi/15.7
Portland MVAPICH2 PGI Fortran version 15.7 4.1.1 3.1 (Nov 2015) Y (32) mvapich2-1.7 1:07:22 UNC module mvapich2_pgi/15.7