CMAQ version 5.3.1 Compiler Test

From CMASWIKI
Revision as of 14:40, 16 December 2019 by Lizadams (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Timing Information for CMAQv5.3.1 - Nov 2019

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3.1 (CMAQv5.3.1) has a planned release date of early December 2019.

Release Testing

The CMAS Center tested the CMAQ release package with the debug and optimized compiler option to compare with the EPA. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. The code used in this testing was obtained from EPA's github account, as a pull request to my github account. https://github.com/lizadams/CMAQ/tree/patches2

I ran using 32 processors for the optimized version.

  @ NPCOL  =  8; @ NPROW =  4


I obtained Fahim's output from atmos and transfered the first day's output to /proj/ie/proj/CMAS/CMAQ/from_EPA/CMAQv531_2016_12SE1_output

Ran for 2 days, GCC OPT version ran successfully for both days.

NOTE: Obtained the following error on the 2nd day of the run using the INTEL compiler 205250 Nov 26 14:03 CTM_LOG_011.v53_intel_Bench_2016_12SE1_full_CONC_20160702

    Processing Day/Time [YYYYDDD:HHMMSS]: 2016184:005500
      Which is Equivalent to (UTC): 0:55:00  Saturday,  July 2, 2016
      Time-Step Length (HHMMSS): 000500
                VDIFF completed...    0.2 seconds
               COUPLE completed...    0.0 seconds
                 HADV completed...    0.1 seconds
                 ZADV completed...    0.0 seconds
                HDIFF completed...    0.0 seconds
             DECOUPLE completed...    0.0 seconds
                 PHOT completed...    0.0 seconds
              CLDPROC completed...    0.1 seconds
                 CHEM completed...    0.4 seconds
                 AERO completed...    0.7 seconds
           Master Time Step
           Processing completed...    1.6 seconds
 MPI_ABORT was invoked on rank 16 in communicator MPI_COMM_WORLD
 with errorcode 538976288.
 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
 You may or may not see output from other processes, depending on
 exactly when Open MPI kills them.
 forrtl: error (78): process killed (SIGTERM)
  Image              PC                Routine            Line        Source
  CCTM_v53.exe       00000000009CB53E  Unknown               Unknown  Unknown
  libpthread-2.17.s  00002B812638D370  Unknown               Unknown  Unknown
  libpthread-2.17.s  00002B812638A213  pthread_spin_lock     Unknown  Unknown
  forrtl: error (78): process killed (SIGTERM)
  Image              PC                Routine            Line        Source


Table 1. CMAQv5.3.1 compilation testing manifest on UNC Dogwood Cluster
Scenario Compiler netCDF I/O API MPI_YN (#P) Module Timing(35 Lay, 220 Var) Notes
Gfortran OpenMPI UNC Debug Gfortran version 9.1.0 4.0.1 3.2 Y (8x4) openmpi_4.0.1_gcc_9.1.0 limit stacksize unlimited
Gfortran OpenMPI EPA Debug Gfortran version 9.1.0 4.0.1 3.2 Y (8x4) openmpi_4.0.1_gcc_9.1.0 3448.4 s/57.47m limit stacksize unlimited
Gfortran Openmpi EPA O3 OPT Gfortran version 9.1.0 4.0.1 3.2 Y (8x4) openmpi_4.0.1_gcc_9.1.0 692.7/11.5m limit stacksize unlimited
Gfortran Openmpi UNC O3 OPT Gfortran version 9.1.0 4.0.1 3.2 Y (8x4) openmpi_4.0.1_gcc_9.1.0 766.83s/12.7m limit stacksize unlimited
Gfortran Openmpi UNC O3 OPT Gfortran version 9.1.0 4.0.1 3.2 Y (4x4) openmpi_4.0.1_gcc_9.1.0 1298.1s/21.6m limit stacksize unlimited
Intel OpenMPI UNC (OPT) Intel version 18.2 4.0.1 3.2 Y (8x4) openmpi_3.1.4_intel_18.2 533.04s/8.84m limit stacksize unlimited

QA plots are generated on longleaf using the interactive queue

R script for comparing model output results from different compilers

  • module load r/3.4.3
  • setenv R_LIBS /nas/longleaf/home/lizadams/R/x86_64-pc-linux-gnu-library/3.4
  • cd /proj/ie/proj/CMAS/CMAQ/CMAQ_v5.3.1_branch2/CMAQ_QA

If you are on the login node, your prompt will contain the login node id:

  • [lizadams@longleaf-login2 ~]

Start an interactive queue (don't run on the login node), your environment variables and R module will be set up

  • srun -p interact --pty /bin/csh

Your login prompt will be something like

  • [lizadams@off03 CMAQ_QA]$

Edit, then Run the R script that compares model output

  • R CMD BATCH plot_max_diff_compiler_sens_v5.3beta.R

Plotting the difference in values between the base (EPA debug) and sensitivity (UNC debug) cases at the column, row, layer of the maximum difference between the variables. | Compilation Testing Plots