View source for WRF-CMAQ v5.3.1 Compiler Test

Latest CMAQv5.3.1 code (CMAQ_REPO_CMAS2way) available from
git clone https://github.com/fisidi/CMAQ.git

Public available version
git clone -b master https://github.com/USEPA/CMAQ.git CMAQ_REPO

Created instructions for how to build and run WRF4.1.1
https://github.com/lizadams/CMAQ/blob/master/DOCS/Users_Guide/Tutorials/CMAQ-WRF_tutorial_build_gcc.md

Note: above instructions include how to download the latest updates from Carlie for ioapi3.2.

Status of runs.

The Runtime error of the debug compiled standalone CMAQ fails with FPE.
/proj/ie/proj/CMAS/WRF-CMAQ/CMAQ_REPO_CMAS2way/openmpi_4.0.1_gcc_9.1.0_debug/CCTM/scripts/slurm-2390504.out
mpirun noticed that process rank 7 with PID 0 on node c-206-11 exited on signal 8 (Floating point exception).


The optimized compiled code ran and created output, but then failed with the following message:
     >>> WARNING in subroutine SHUT3 <<<
     Error closing netCDF file
     File name:  CTM_CONC_1
     netCDF error number  -33


      *** FATAL ERROR shutting down Models-3 I/O ***
     The elapsed time for this simulation was    1337.2 seconds.

I was able to create plots using the output from the optimized version and compare it to Fahim's optimized compile of the CMAQ_REPO_CMAS2way

https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/CMAQv5.3.1_fahim/plots/compiler_sens/base_from_fahim/layer1_only

These show agreement between the two.
Fahim provided two sets of outputs, one with ioapi-3.2 older, and one with the latest release.
I downloaded and used m3diff, and found no difference between the ACONC file output for the difference between the CCTM_v531 using the CMAQ_REPO_CMAS2way

     FILE A:  AFILE (./data_04_19_2020/output_CCTM_v531_gcc9.1_Bench_2016_12SE1/CCTM_ACONC_v531_gcc9.1_Bench_2016_12SE1_20160701.nc)

     FILE B:  BFILE (./data/output_CCTM_v531_gcc9.1_Bench_2016_12SE1/CCTM_ACONC_v531_gcc9.1_Bench_2016_12SE1_20160701.nc)

Date and time  2016183:000000 (0:00:00   July 1, 2016)
 A:AFILE/NO2  vs  B:BFILE/NO2  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    1.92732E-02@( 31,39, 1)  2.38787E-05@( 94,13, 1)  1.23437E-03  1.58040E-03
 B    1.92732E-02@( 31,39, 1)  2.38787E-05@( 94,13, 1)  1.23437E-03  1.58040E-03
 A:B  0.00000E+00@(  1, 0, 0)  0.00000E+00@(  1, 0, 0)  0.00000E+00  0.00000E+00


I also found no difference between my run and Fahim's

     FILE A:  AFILE (data/output_CCTM_v531_gcc9.1_Bench_2016_12SE1/CCTM_ACONC_v531_gcc9.1_Bench_2016_12SE1_20160701.nc)

     FILE B:  BFILE (/proj/ie/proj/CMAS/WRF-CMAQ/CMAQ_REPO_CMAS2way/openmpi_4.0.1_gcc_9.1.0/data/output_CCTM_v531_gcc_Bench_2016_12SE1_error/CCTM_ACONC_v531_gcc_Bench_2016_12SE1_20160701.nc)

 Date and time  2016183:000000 (0:00:00   July 1, 2016)
 A:AFILE/NO2  vs  B:BFILE/NO2  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    1.92732E-02@( 31,39, 1)  2.38787E-05@( 94,13, 1)  1.23437E-03  1.58040E-03
 B    1.92732E-02@( 31,39, 1)  2.38787E-05@( 94,13, 1)  1.23437E-03  1.58040E-03
 A:B  0.00000E+00@(  1, 0, 0)  0.00000E+00@(  1, 0, 0)  0.00000E+00  0.00000E+00

So, we are getting matching answers for the optimized CMAQv5.3.1 latest code.

Fahim obtained matching answers for WRF-CMAQ using new and old ioapi-3.2.

     FILE A:  AFILE (data/output_12km_nf_rrtmg_20_5_1_v411531/CCTM_ACONC_v411531_20160701.nc)

     FILE B:  BFILE (data_04_19_2020/output_12km_nf_rrtmg_20_5_1_v411531/CCTM_ACONC_v411531_20160701.nc)

 Date and time  2016183:000000 (0:00:00   July 1, 2016)
 A:AFILE/NO2  vs  B:BFILE/NO2  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    2.22875E-02@(  9,37, 1)  2.36311E-05@( 95,13, 1)  1.29369E-03  1.71342E-03
 B    2.22875E-02@(  9,37, 1)  2.36311E-05@( 95,13, 1)  1.29369E-03  1.71342E-03
 A:B  0.00000E+00@(  1, 0, 0)  0.00000E+00@(  1, 0, 0)  0.00000E+00  0.00000E+00


When I use m3diff to compare my WRF-CMAQ and Fahim's results, I am seeing differences at the first timestep.



     FILE A:  AFILE (/proj/ie/proj/CMAS/WRF-CMAQ/openmpi_4.0.1_gcc_9.1.0_debug/WRF-4.1.1/output_12km_nf_rrtmg_20_5_1_v41153_2016/CCTM_CONC_v41153_20160701.nc)

     FILE B:  BFILE (/proj/ie/proj/CMAS/WRF-CMAQ/from_fahim/data/output_12km_nf_rrtmg_20_5_1_v411531_debug/CCTM_CONC_v411531_20160701.nc)


     -----------------------------------------------------------




 Date and time  2016183:000000 (0:00:00   July 1, 2016)
 A:AFILE/O3  vs  B:BFILE/O3  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    8.12574E-02@( 35,46, 1)  1.31113E-02@( 53, 4, 1)  3.55808E-02  1.00917E-02
 B    8.12574E-02@( 35,46, 1)  1.31113E-02@( 53, 4, 1)  3.55808E-02  1.00917E-02
 A:B  0.00000E+00@(  1, 0, 0)  0.00000E+00@(  1, 0, 0)  0.00000E+00  0.00000E+00


 Date and time  2016183:010000 (1:00:00   July 1, 2016)
 A:AFILE/O3  vs  B:BFILE/O3  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    8.16709E-02@( 35,48, 1)  1.18036E-02@( 36,10, 1)  3.20051E-02  8.83139E-03
 B    7.84062E-02@( 35,48, 1)  1.11254E-02@( 59,28, 1)  3.18859E-02  8.73194E-03
 A:B  4.59358E-03@( 34,48, 1) -1.09005E-02@( 30,41, 1)  1.19151E-04  6.41393E-04


 Date and time  2016183:020000 (2:00:00   July 1, 2016)
 A:AFILE/O3  vs  B:BFILE/O3  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    7.22206E-02@( 35,50, 1)  6.84631E-03@( 59,28, 1)  2.91940E-02  8.49730E-03
 B    7.64798E-02@( 35,49, 1)  4.48307E-03@( 59,28, 1)  2.91144E-02  8.45926E-03
 A:B  7.65019E-03@( 36,50, 1) -1.17895E-02@( 30,41, 1)  7.96838E-05  6.51846E-04

--


     FILE A:  AFILE (data/output_12km_nf_rrtmg_20_5_1_v411531/CCTM_ACONC_v411531_20160701.nc)

     FILE B:  BFILE (/proj/ie/proj/CMAS/WRF-CMAQ/openmpi_4.0.1_gcc_9.1.0/WRF-4.1.1/output_12km_nf_rrtmg_20_5_1_v41153_2016/CCTM_ACONC_v41153_20160701.nc)


 Date and time  2016183:000000 (0:00:00   July 1, 2016)
 A:AFILE/NO2  vs  B:BFILE/NO2  vs  (A - B)
      MAX        @(  C, R, L)  Min        @(  C, R, L)  Mean         Sigma
 A    2.22875E-02@(  9,37, 1)  2.36311E-05@( 95,13, 1)  1.29369E-03  1.71342E-03
 B    2.22726E-02@(  9,37, 1)  2.36312E-05@( 95,13, 1)  1.29924E-03  1.73567E-03
 A:B  1.26635E-03@( 59,28, 1) -5.31193E-03@( 31,39, 1) -5.55327E-06  1.42523E-04

You can see the difference plots in the following link.
https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/WRF-CMAQ/fahim_base/plots/compiler_sens/base_from_fahim/layer1_only

I obtained the run scripts that Fahim was using on atmos.  With the updated run scripts, I am able to match his results for the debug version, and my optimized version matches Fahim's version.

Here are the updated plots
https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/WRF-CMAQ/fahim_debug_base/plots/compiler_sens/base_from_fahim/layer1_only

Turned on the sf option for the code from Fahim and obtained matching answers for the debug version.

Here are the plots
https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/WRF-CMAQ/wrf_cmaq_sf/plots/compiler_sens/base_from_fahim/layer1_only


Older comparisons:


Plotting the difference in values between the base (EPA gcc_ioapi_3.2_epa) and sensitivity (UNC gcc ioapi_3.2_epa, UNC intel ioapi_3.2_unc, UNC intel ioapi_3.2_epa ) plot of value of each species at the column, row, layer of the maximum difference between the variables. 
[https://dataviewer-dept-cempd.cloudapps.unc.edu/compare.cfm?back_address1=/WRF-CMAQ/epa_base/plots/compiler_sens/base_from_EPA/layer1_only&back_address2=/WRF-CMAQ/epa_base/plots/prc_diff/base_from_EPA/layer1_only | Compilation Testing Plots]