CMAQ version 5.3beta UNC4 orig Technical Documentation

From CMASWIKI
Revision as of 11:04, 2 July 2019 by Lizadams (talk | contribs) (Created page with "= Timing Information for CMAQv5.3beta_UNC4 - May 2019 = The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3 (CMAQv5.3) is scheduled for release in July 2...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Timing Information for CMAQv5.3beta_UNC4 - May 2019

The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3 (CMAQv5.3) is scheduled for release in July 2019.
To facilitate community contributions to this model release, a “CMAQv5.3_branch_v53_UG” version was created and made available to external collaborators/developers in May 2019. This Beta_UNC4-version will undergo extensive application and evaluation over the next few months, which may lead to additional changes before the final release.

The CMAQv5.3_branch_v53_UG repository is available from https://github.com/kmfoley/CMAQ/tree/v53_UG.

Updated documentation for CMAQv5.3beta, including build instructions and technical features are available as well.

Release Testing

The CMAS Center will test the CMAQv5.3_branch_v53_UG release package with several versions and configurations of the Portland Group, Intel, and GNU Fortran compilers. In addition to different serial and parallel configurations, the release will be tested under different science configurations. The results of the CMAS Center tests will include run times for different compilers and configurations are provided below. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. This dataset is distributed for benchmarking the model installation and includes new EPIC files.

Two classes of tests:

  • Compiler tests used the default benchmark configuration with different compilers and MPI configurations.
  • Configuration tests used the gcc or intel vXX OpenMPI compiler to generate executables that exercise different scientific configurations of the release software.

Compiler flags:

  • GCC: mpif90 -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising
  • Intel: mpifort -c -fixed -132 -O3 -fno-alias -mp1 -fp-model source -ftz -simd -align all -axCOMMON-AVX512 -vec-guard-write -unroll-aggressive -mssse3
  • Link step:

mpifort -qopenmp-simd *.o -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_v53_UG/openmpi_3.1.4_intel_18.2/lib/x86_64/intel/ioapi/lib -lioapi -L/proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_v53_UG/openmpi_3.1.4_intel_18.2/lib/x86_64/intel/netcdf/lib -lnetcdff -lnetcdf -L /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_v53_UG/openmpi_3.1.4_intel_18.2/lib/x86_64/intel/pnetcdf/lib -lpnetcdf -qopenmp -o CCTM_v53.exe
Note: this website contains explanation of why timing varies depending on the mpi flavor that is being used, and suggests flags to pin or bind tasks to a core set of cpus,and also options to use the infiniband. https://www.chpc.utah.edu/documentation/software/mpilibraries.php http://www.nersc.gov/users/computational-systems/cori/running-jobs/example-batch-scripts/

Specs for Dogwood machine: https://help.unc.edu/files/2018/03/dogwood.cheatsheet.pdf


Table 1. CMAQv5.3beta compilation testing manifest on UNC Dogwood Cluster
Scenario Compiler netCDF I/O API MPI_YN (#P) Module Timing(1 Lay, 12 Var)cpu/wall/ratio Timing(35 Lay, 12 Var) Timing (35 Lay, 215 Var) Notes
Gfortran OpenMPI Gfortran version 6.3 4.6.2 3.2 Y (64) openmpi_3.0.0/gcc_6.3.0 time mpirun --oversubscribe -np $NPROCS, using srun didn't work
Gfortran MPICH Gfortran version 4.8.1 4.6.0 3.2 Y (64) mpich/3.2.1_1 Linuxbrew gcc (it is performing like it is running on 1 processor..)
Intel OpenMPI Intel Fortran version 17.2 4.1.3 3.2 Y (4) openmpi_3.0.0/intel_17.2 13326/3388/3.93
Intel OpenMPI Intel Fortran version 17.2 4.1.3 3.2 Y (16) openmpi_3.0.0/intel_17.2 16090/1016/15/83
Intel OpenMPI Intel Fortran version 17.2 4.1.3 3.2 Y (64) openmpi_3.0.0/intel_17.2 37868/1465.52/25.8
Intel OpenMPI Intel Fortran version 17.2 4.1.3 3.2 Y (128) openmpi_3.0.0/intel_17.2
Intel OpenMPI Intel Fortran version 17.2 4.1.3 3.2 Y (256) openmpi_3.0.0/intel_17.2
Intel OpenMPI Intel Fortran version 18.2 4.6.2 3.2 Y (16) openmpi_3.1.4/intel_18.2 15729/991/15.8
Intel OpenMPI Intel Fortran version 18.2 4.6.2 3.2 Y (32) openmpi_3.1.4/intel_18.2 47948/1851/25.9
Intel OpenMPI (limit stacksize unlimited) Intel Fortran version 18.2 4.6.2 3.2 Y (32) openmpi_3.1.4/intel_18.2 8257/522/15.8
Intel OpenMPI Intel Fortran version 18.2 4.6.2 3.2 Y (64) openmpi_3.1.4/intel_18.2 37446/1447/25.8
Intel OpenMPI (stacksize unlimited) Intel Fortran version 18.2 4.6.2 3.2 Y (64) openmpi_3.1.4/intel_18.2 39237/1544.9/25.4
Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive) Intel Fortran version 18.2 4.6.2 3.2 Y (16) openmpi_3.1.4/intel_18.2 15233/958/15.9
Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive) Intel Fortran version 18.2 4.6.2 3.2 Y (32) openmpi_3.1.4/intel_18.2 17540/585/29.9
Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive) Intel Fortran version 18.2 4.6.2 3.2 Y (64) openmpi_3.1.4/intel_18.2 10126/349/29.0
Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive) Intel Fortran version 18.2 4.6.2 3.2 Y (64) openmpi_3.1.4/intel_18.2 9345/340/27.38
Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive) Intel Fortran version 18.2 4.6.2 3.2 Y (128) openmpi_3.1.4/intel_18.2 5259/179/29.3
Intel OpenMPI (stacksize unlimited and #SBATCH --exclusive) Intel Fortran version 18.2 4.6.2 3.2 Y (256) openmpi_3.1.4/intel_18.2 3495/121/28.9
Intel MVAPICH2 Intel Fortran version 17.2 4.1.3 3.2 Y (64) mvapich2_2.3rc1/intel_17.2 setenv MV2_ENABLE_AFFINITY 0 see:https://github.com/USEPA/CMAQ/issues/118
Intel MVAPICH2 Intel Fortran version 17.2 4.1.3 3.2 Y (128) mvapich2_2.3rc1/intel_17.2 setenv MV2_ENABLE_AFFINITY 0 see:https://github.com/USEPA/CMAQ/issues/118
Intel MVAPICH2 Intel Fortran version 17.2 4.1.3 3.2 Y (256) mvapich2_2.3rc1/intel_17.2 setenv MV2_ENABLE_AFFINITY 0 see:https://github.com/USEPA/CMAQ/issues/118
Table 2. CMAQv5.3beta configuration testing manifest
Scenario Description Mechanism Notes Timing (64 PE)
Benchmark Case Online emissions processing, inline photolysis, inline lightning using hourly NLDN strike file, windblown dust, surface HONO, bidirectional NH3, no BIDI HG, no potential vorticity scaling cb6r3_ae7_aq Pending; LTNGNO InLine, LTNGPARM = N, LOG_START = 2.0
Halogen Chemistry Same as Benchmark case with halogen chemistry enabled cb6r3_ae7_aq Pending. Turned off the diagnostic file logging.
No Bidi Same as Benchmark with Hg and NH3 BiDi deactivated cb6r3_ae7_aq Pending. set CTM_HGBIDI = N; set ABFLUX = N
Process Analysis Benchmark case with IPR and IRR ros3 Pending. Switch to Rosenbrock solver because EBI solver not supported by PA module; ran with inline process analysis
MOSAIC Benchmark case with MOSAIC and additional stomatal flux files activated cb6r3_ae7_aq Pending. set CTM_MOSAIC = Y; set CTM_FST = Y
New Mechanism Test Benchmark case with toluene and chlorine chemistry cb05tucl_ae6 Pending.
Potential vorticity UTLS exchange Benchmark case with scaling for STE cb6r3_ae7_aq Pending: Uncomment potvortO3 in build. Need PV variable in METCRO3D file
Dust Benchmark case with dust, including new MODIS FP input cb6r3_ae7_aq Pending. setenv CTM_WB_DUST Y; setenv CTM_ERODE_AGLAND Y; setenv CTM_WBDUST_BELD BELD3
Hourly NLDN Benchmark with lightning NOx calculated using hourly bNLDN strikes cb6r3_ae7_aq Pending; LTNGNO InLine, LTNGPARM = Y, USE_NLDN Y
POA Sensitivity Benchmark with new POA mechanism cb6r3_ae7_aq #set potvortO3 #> potential vorticity from the free troposphere

QA plots are generated on longleaf using the interactive queue

R script for comparing model output results from different compilers

  • module load r/3.4.3
  • setenv R_LIBS /nas/longleaf/home/lizadams/R/x86_64-pc-linux-gnu-library/3.4
  • cd /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC3/CMAQ_QA

If you are on the login node, your prompt will contain the login node id:

  • [lizadams@longleaf-login2 ~]

Start an interactive queue (don't run on the login node), your environment variables and R module will be set up

  • srun -p interact --pty /bin/csh

Your login prompt will be something like

  • [lizadams@off03 CMAQ_QA]$

Run the R script that compares model output

  • R CMD BATCH plot_max_diff_compiler_sens_v5.3beta.R

Plotting the difference in values between the base and sensitivity cases at the column, row, layer of the maximum difference between the variables. | Compilation Testing Plots

Retrieved from "https://www.airqualitymodeling.org/index.php?title=CMAQ_version_5.3beta_UNC4_orig_Technical_Documentation&oldid=1482"