View source for CMAQ version 5.3beta UNC6 (July 2019 release) Technical Documentation

= Timing Information for CMAQv5.3beta_UNC6 - July 2019  =

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3 (CMAQv5.3) is scheduled for release in July 2019. <br>To facilitate community contributions to this model release, a “cmaqv53070119.tar.gz” version was created by Fahim. 

Updated documentation for CMAQv5.3beta, including build instructions and technical features are available as well.

= Release Testing =

The CMAS Center will test the cmaqv53070119.tar.gz release package with several versions and configurations of the Portland Group, Intel, and GNU Fortran compilers.  In addition to different serial and parallel configurations, the release will be tested under different science configurations. The results of the CMAS Center tests will include run times for different compilers and configurations are provided below. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. This dataset is distributed for benchmarking the model installation and includes new EPIC files. 

Two classes of tests:  
*''Compiler'' tests used the default benchmark configuration with different compilers and MPI configurations.  
*''Configuration'' tests used the gcc or intel vXX OpenMPI compiler to generate executables that exercise different scientific configurations of the release software. 

Compiler flags:
* GCC DEBUG: mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/include_files -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi -I. se_bndy_copy_info_ext.f
* GCC NODEBUG: mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512 -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/include_files -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi -I. se_bndy_copy_info_ext.f
* Link step: 
<code>
mpifort se_bndy_copy_info_ext.o .. subhfile.o -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -lioapi -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/netcdf/lib -lnetcdf -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/netcdff/lib -lnetcdff  -o CCTM_v53.exe
</code>
<br>
Note: this website contains explanation of why timing varies depending on the mpi flavor that is being used, and suggests flags to pin or bind tasks to a core set of cpus,and also options to use the infiniband.
https://www.chpc.utah.edu/documentation/software/mpilibraries.php
http://www.nersc.gov/users/computational-systems/cori/running-jobs/example-batch-scripts/

Specs for Dogwood machine: https://help.unc.edu/files/2018/03/dogwood.cheatsheet.pdf<br>


{| class="wikitable"
|+ Table 1. CMAQv5.3beta compilation testing manifest on UNC Dogwood Cluster
|-
!Scenario !! Compiler !! netCDF !! I/O API !! MPI_YN (#P) !! Module !! Timing(1 Lay, 12 Var)real/user/system!!Timing(35 Lay, 12 Var)!!Timing (35 Lay, 215 Var)!! Notes
|
|-
|Gfortran OpenMPI Debug
|Gfortran version 9.1.0
|4.0.1
|3.2
|Y (32)
|openmpi_4.0.1_gcc_9.1.0
|failed with FPE
|
|limit stacksize unlimited
|time mpirun -np $NPROCS
|-
|Gfortran OpenMPI
|Gfortran version 9.1.0
|4.0.1
|3.2
|Y (32)
|openmpi_4.0.1_gcc_9.1.0
|
|
|limit stacksize unlimited
|time mpirun -np $NPROCS
|-
|Gfortran OpenMPI
|Gfortran version 9.1.0
|4.0.1
|3.2
|Y (64)
|openmpi_4.0.1_gcc_9.1.0
|13537.34/464.48/287.85
|
|
|limit stacksize unlimited
|-
|Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive)
|Intel Fortran version 18.2
|4.6.2
|3.2
|Y (32)
|openmpi_3.1.4/intel_18.2
|
|
|17540/585/29.9
|
|-
|
|}

QA plots are generated on longleaf using the interactive queue

R script for comparing model output results from different compilers
* module load r/3.4.3
* setenv R_LIBS /nas/longleaf/home/lizadams/R/x86_64-pc-linux-gnu-library/3.4
* cd /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC3/CMAQ_QA

If you are on the login node, your prompt will contain the login node id:
* [lizadams@longleaf-login2 ~]

Start an interactive queue (don't run on the login node), your environment variables and R module will be set up
* srun -p interact --pty /bin/csh

Your login prompt will be something like
* [lizadams@off03 CMAQ_QA]$

Run the R script that compares model output
* R CMD BATCH plot_max_diff_compiler_sens_v5.3beta.R

Plotting the difference in values between the base and sensitivity cases at the column, row, layer of the maximum difference between the variables.
[https://dataviewer-dept-cempd.cloudapps.unc.edu/compare.cfm?back_address1=/CMAQv5.3b_UNC4_QA/compiler_sens/base_openmpi_3.1.4_intel_18.2_16pe/all_layers&back_address2=/CMAQv5.3b_UNC4_QA/compiler_sens/base_openmpi_3.1.4_intel_18.2_16pe/layer1_only | Compilation Testing Plots]

= Release Testing =

The CMAS Center will test the CMAQv5.3_branch_v53_20190613  release package with several versions and configurations of the Portland Group, Intel, and GNU Fortran compilers.  In addition to different serial and parallel configurations, the release will be tested under different science configurations. The results of the CMAS Center tests will include run times for different compilers and configurations are provided below. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. This dataset is distributed for benchmarking the model installation and includes new EPIC files. 

Two classes of tests:  
*''Compiler'' tests used the default benchmark configuration with different compilers and MPI configurations.  
*''Configuration'' tests used the gcc or intel vXX OpenMPI compiler to generate executables that exercise different scientific configurations of the release software. 

Compiler flags:
* GCC DEBUG: mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/include_files -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi -I. se_bndy_copy_info_ext.f
* GCC NODEBUG: mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512 -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/include_files -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi -I. se_bndy_copy_info_ext.f
* Link step: 
<code>
mpifort se_bndy_copy_info_ext.o .. subhfile.o -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -lioapi -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/netcdf/lib -lnetcdf -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_fahim_UNC6/cmaqv53070119/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/netcdff/lib -lnetcdff  -o CCTM_v53.exe
</code>
<br>
Note: this website contains explanation of why timing varies depending on the mpi flavor that is being used, and suggests flags to pin or bind tasks to a core set of cpus,and also options to use the infiniband.
https://www.chpc.utah.edu/documentation/software/mpilibraries.php
http://www.nersc.gov/users/computational-systems/cori/running-jobs/example-batch-scripts/

Specs for Dogwood machine: https://help.unc.edu/files/2018/03/dogwood.cheatsheet.pdf<br>


{| class="wikitable"
|+ Table 1. CMAQv5.3beta compilation testing manifest on UNC Dogwood Cluster
|-
!Scenario !! Compiler !! netCDF !! I/O API !! MPI_YN (#P) !! Module !! Timing(1 Lay, 12 Var)real/user/system!!Timing(35 Lay, 12 Var)!!Timing (35 Lay, 215 Var)!! Notes
|
|-
|Gfortran OpenMPI Debug
|Gfortran version 9.1.0
|4.0.1
|3.2
|Y (32)
|openmpi_4.0.1_gcc_9.1.0
|failed with FPE
|
|limit stacksize unlimited
|time mpirun -np $NPROCS
|-
|Gfortran OpenMPI
|Gfortran version 9.1.0
|4.0.1
|3.2
|Y (32)
|openmpi_4.0.1_gcc_9.1.0
|
|
|limit stacksize unlimited
|time mpirun -np $NPROCS
|-
|Gfortran OpenMPI
|Gfortran version 9.1.0
|4.0.1
|3.2
|Y (64)
|openmpi_4.0.1_gcc_9.1.0
|13537.34/464.48/287.85
|
|
|limit stacksize unlimited
|-
|Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive)
|Intel Fortran version 18.2
|4.6.2
|3.2
|Y (32)
|openmpi_3.1.4/intel_18.2
|
|
|17540/585/29.9
|
|-
|
|}

QA plots are generated on longleaf using the interactive queue

R script for comparing model output results from different compilers
* module load r/3.4.3
* setenv R_LIBS /nas/longleaf/home/lizadams/R/x86_64-pc-linux-gnu-library/3.4
* cd /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC3/CMAQ_QA

If you are on the login node, your prompt will contain the login node id:
* [lizadams@longleaf-login2 ~]

Start an interactive queue (don't run on the login node), your environment variables and R module will be set up
* srun -p interact --pty /bin/csh

Your login prompt will be something like
* [lizadams@off03 CMAQ_QA]$

Run the R script that compares model output
* R CMD BATCH plot_max_diff_compiler_sens_v5.3beta.R

Plotting the difference in values between the base and sensitivity cases at the column, row, layer of the maximum difference between the variables.
[https://dataviewer-dept-cempd.cloudapps.unc.edu/compare.cfm?back_address1=/CMAQv5.3b_UNC4_QA/compiler_sens/base_openmpi_3.1.4_intel_18.2_16pe/all_layers&back_address2=/CMAQv5.3b_UNC4_QA/compiler_sens/base_openmpi_3.1.4_intel_18.2_16pe/layer1_only | Compilation Testing Plots]