Difference between revisions of "CMAQv532 on C5.4xlarge"

From CMASWIKI
Jump to: navigation, search
Line 104: Line 104:
  
 
The version of openmpi on the Amazon EC2 instance is
 
The version of openmpi on the Amazon EC2 instance is
 +
```
 
[cmas@ip-172-31-92-184 Scripts-CMAQ]$ mpirun --version
 
[cmas@ip-172-31-92-184 Scripts-CMAQ]$ mpirun --version
 
mpirun (Open MPI) 4.0.3
 
mpirun (Open MPI) 4.0.3
 
gcc --version
 
gcc --version
 
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
 
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
 +
```

Revision as of 20:38, 14 January 2021

Amazon AMI EC2 Instance: C5.4xlarge (16 processors)

openmpi_4.0.1/gcc_8.3.1

==================================
  ***** CMAQ TIMING REPORT *****
==================================
Start Day: 2016-07-01
End Day:   2016-07-02
Number of Simulation Days: 2
Domain Name:               2016_12SE1
Number of Grid Cells:      280000  (ROW x COL x LAY)
Number of Layers:          35
Number of Processes:       16
   All times are in seconds.

Num  Day        Wall Time
01   2016-07-01   1730.5
02   2016-07-02   1602.3
     Total Time = 3332.80
      Avg. Time = 1666.40

     The elapsed time for this simulation was    1602.3 seconds.

19711.615u 1046.727s 26:42.77 1295.1%   0+0k 6735848+1416040io 6pf+0w

CMAQ Processing of Day 20160702 Finished at Wed Dec 16 18:47:51 UTC 2020

Singularity mvapich
Note singularity CMAQ CCTM uses the medium memory model.

 X86_64 "Medium memory model" version:  support stack-size,
#  array-size, data-size larger than 2 GB.
#  Use of this opotion requires that "gcc" and "gfortran" thenselves be
#  of version 4.4 or later and have been compiled with  "-mcmodel=medium".
#  See http://eli.thegreenplace.net/2012/01/03/understanding-the-x64-code-models

Num    Day         Wall Time
01     2016-07-01   1546.4 
02     2016-07-01   1468.0
Total Time = 3015.33
Avg. Time = 1507.66


Run Times on 16 pe on c5.4xlarge
CMAQv5.3.2 (openmpi) CMAQv5.3.2 (mpich) CMAQv5.3.2 Singularity (openmpi) CMAQv5.3.2 Singularity (mvapich) CMAQv5.3.2 Singularity (mpich) CMAQv5.3.2 Singularity Atmos (openmpi)
day 1 1730.5 1779.1 error 1546.4 1564.5 1151
day 2 1602.3 1649.6 error 1468.0 1497.33

Error for openmpi on C5.4xlarge with attempt to run on 16 processors /usr/bin/time -p mpirun -np 16 /opt/CMAQ_532/CCTM/scripts/BLD_CCTM_v532_gcc-openmpi/CCTM_v532.exe [1610650809.265129] [ip-172-31-84-61:6018 :0] sys.c:618 UCX ERROR shmget(size=2097152 flags=0xfb0) for mm_recv_desc failed: Operation not permitted, please check shared memory limits by 'ipcs -l'

       CTM_APPL  |  v532_openmpi_gcc_2016_12SE1_20160701

Then it only creates a run on 1 processor with one log file:

***  ERROR in INIT3/INITLOG3  ***
    Error opening log file on unit        99
    I/O STATUS =        17
    DESCRIPTION: File 'CTM_LOG_000.v532_openmpi_gcc_2016_12SE1_20160701' already exists
    File: CTM_LOG_000.v532_openmpi_gcc_2016_12SE1_20160701

[1610650809.275238] [ip-172-31-84-61:6012 :0] sys.c:618 UCX ERROR shmget(size=2097152 flags=0xfb0) for mm_recv_desc failed: Operation not permitted, please check shared memory limits by 'ipcs -l'

top shows that the job is running on just 1 processor top - 19:05:25 up 50 min, 2 users, load average: 1.48, 0.65, 1.04 Tasks: 229 total, 3 running, 211 sleeping, 0 stopped, 15 zombie %Cpu(s): 8.9 us, 3.5 sy, 0.0 ni, 87.4 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st MiB Mem : 31157.2 total, 24247.2 free, 5918.4 used, 991.5 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 24847.4 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                            
  6160 cmas      20   0   25324   2468   2180 R  99.7   0.0   1:17.64 hydra_pmi_proxy                                                    
  6161 cmas      20   0 6415232   5.5g  19592 R  99.3  18.0   1:17.63 CCTM_v532.exe                                                      
  6219 cmas      20   0   65520   4800   3916 R   0.3   0.0   0:00.01 top                                                                
     1 root      20   0  244840  13440   9088 S   0.0   0.0   0:03.11 systems

The version of openmpi on the Amazon EC2 instance is ``` [cmas@ip-172-31-92-184 Scripts-CMAQ]$ mpirun --version mpirun (Open MPI) 4.0.3 gcc --version gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5) ```