CMAQ version 5.3release UNC10 Compiler Test

From CMASWIKI
Revision as of 14:59, 12 September 2019 by Lizadams (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Timing Information for CMAQv5.3beta_UNC10 - September 2019

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3 (CMAQv5.3) was released in Aug. 2019.

Release Testing

The CMAS Center tested the CMAQ release package with the debug and optimized compiler option to compare with the EPA. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. The code used in this testing was obtained from EPA's github account, as a pull request to my github account. https://github.com/lizadams/CMAQ/pull/13

Fahim requested the configuration of how many processors I was running on. I ran using 8 processors for the optimized version.

  @ NPCOL  =  2; @ NPROW =  4

I tried running with 8 processors for the debug version, but it doesn't finish in 4 hours, so I am currently running the debug version on 16 processors.

  @ NPCOL  =  4; @ NPROW =  4


I obtained Fahim's output from atmos and transfered the first day's output to /proj/ie/proj/CMAS/CMAQ/from_EPA/output_CCTM_v53_gcc9.1_Bench_2016_12SE1 /proj/ie/proj/CMAS/CMAQ/from_EPA/output_CCTM_v53_gcc9.1_Bench_2016_12SE1_debug

Two classes of tests:

  • Compiler tests used the default benchmark configuration with the GNU debug compiler.

Compiler flags:

  • GCC DEBUG: mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/include_files -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi -I. -Dparallel -Dm3dry_opt -DSUBST_BARRIER=SE_BARRIER -DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX -DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=SE_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=SE_LOOP_INDEX -DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE -DSUBST_SUM_CHK=SE_SUM_CHK -DSUBST_INIT_ARRAY=SE_INIT_ARRAY -DSUBST_COMM=SE_COMM -DSUBST_MY_REGION=SE_MY_REGION -DSUBST_SLICE=SE_SLICE -DSUBST_GATHER=SE_GATHER -DSUBST_DATA_COPY=SE_DATA_COPY -DSUBST_IN_SYN=SE_IN_SYN -DSUBST_PE_COMM=\"./PE_COMM.EXT\" -DSUBST_CONST=\"./CONST.EXT\" -DSUBST_FILES_ID=\"./FILES_CTM.EXT\" -DSUBST_EMISPRM=\"./EMISPRM.EXT\" -DSUBST_MPI=\"/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi/include/mpif.h\" subhfile.F
  • Link step:

mpifort *.o -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib -lioapi -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/netcdf/lib -lnetcdff -lnetcdf -o CCTM_v53.exe

GCC O3 OPT:


Table 1. CMAQv5.3beta compilation testing manifest on UNC Dogwood Cluster
Scenario Compiler netCDF I/O API MPI_YN (#P) Module Timing(35 Lay, 220 Var) Notes
Gfortran OpenMPI UNC Debug Gfortran version 9.1.0 4.0.1 3.2 Y (4x4) openmpi_4.0.1_gcc_9.1.0 7846.8s/130.7m limit stacksize unlimited
Gfortran OpenMPI EPA Debug Gfortran version 9.1.0 4.0.1 3.2 Y (4x4) openmpi_4.0.1_gcc_9.1.0 6420.5s/107m limit stacksize unlimited
Gfortran Openmpi UNC O3 OPT Gfortran version 9.1.0 4.0.1 3.2 Y (2x4) openmpi_4.0.1_gcc_9.1.0 2533.8s/42.23m limit stacksize unlimited
Gfortran Openmpi EPA O3 OPT Gfortran version 9.1.0 4.0.1 3.2 Y (2x4) openmpi_4.0.1_gcc_9.1.0 2094.3s/34.9m limit stacksize unlimited
Gfortran Openmpi UNC O3 OPT Gfortran version 9.1.0 4.0.1 3.2 Y (4x4) openmpi_4.0.1_gcc_9.1.0 1364.1s/22.7m limit stacksize unlimited

QA plots are generated on longleaf using the interactive queue

R script for comparing model output results from different compilers

  • module load r/3.4.3
  • setenv R_LIBS /nas/longleaf/home/lizadams/R/x86_64-pc-linux-gnu-library/3.4
  • cd /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC3/CMAQ_QA

If you are on the login node, your prompt will contain the login node id:

  • [lizadams@longleaf-login2 ~]

Start an interactive queue (don't run on the login node), your environment variables and R module will be set up

  • srun -p interact --pty /bin/csh

Your login prompt will be something like

  • [lizadams@off03 CMAQ_QA]$

Run the R script that compares model output

  • R CMD BATCH plot_max_diff_compiler_sens_v5.3beta.R

Plotting the difference in values between the base (EPA debug) and sensitivity (UNC debug) cases at the column, row, layer of the maximum difference between the variables. | Compilation Testing Plots


Found an option to add information to an executable to allow the user to understand what compile options and libraries were used to build it.


Another idea is to have users add the

-frecord-gcc-switches option to the compile flags for the GNU compiler:


I tried evaluating the CCTM

readelf -p .GCC.command.line CCTM_v53.exe


It may prove to be useful when we are trying to help people understand how their executable was compiled.


``` String dump of section '.GCC.command.line':

 [     0]  se_bndy_copy_info_ext.f
 [    18]  -ffixed-form
 [    25]  -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/lib
 [    86]  -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/ioapi/include_files
 [    f1]  -I /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC8/openmpi_4.0.1_gcc_9.1.0/lib/x86_64/gcc/mpi
 [   14c]  -I .
 [   151]  -I /nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/include
 [   193]  -I /nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/lib
 [   1d1]  -mtune=generic
 [   1e0]  -march=x86-64
 [   1ee]  -g
 [   1f1]  -O0
 [   1f5]  -ffixed-line-length-132
 [   20d]  -funroll-loops
 [   21c]  -finit-character=32
 [   230]  -frecord-gcc-switches
 [   246]  -fintrinsic-modules-path /nas/longleaf/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/finclude

```



I found these instructions from the following link:

https://stackoverflow.com/questions/12112338/get-the-compiler-options-from-a-compiled-executable