CMAQv5.0 Parallel notes

PARALLEL_NOTES - 30 September 2008

Jeff Young

The following are some observations related to running the CMAQ CCTM in a Linux MPICH cluster environment where the various computational nodes are connected via Network File System (NFS).

Running the CCTM for the 36km benchmark test case on a parallel cluster with 8 processors took 4.7 minutes wall time, whereas running it on a single processor took 14.5 minutes. The reason for the poor scaling of the parallel run has to do with the size of the problem (domain) in relation to the proportion of I/O processing, which is inherently serial. In addition, the relative inefficiency of NFS in sending internet packets across the local network fast enough to keep the CPUs on the participating nodes busy may contribute.

Clusters that have local high-bandwidth interconnects or are isolated from general network traffic (e.g. Scyld Beowulf) will not suffer this problem as much. That is, even relatively small problem sizes will still go faster in parallel than in serial.

We infrequently experienced the situation where a run on 8 processors (one local, 3 remote dual cpu Intel XEON boxes) would start and hang. But re-launching the run succeeded. This may be due to network latency problems.

We had difficulties running in the MPICH cluster associated with the standard practice of automouting directories. It was necessary to "hard mount" the data directory to which we wrote the CMAQ outputs.

To make our MPICH linux cluster work, we had to put all the machines that we wanted to use in the ~/.rhosts file. Otherwise, we would get a "permission denied" when you launch mpirun.

One operational note: In addition to the main log file, produced by the "processor 0" task, the runs produce ancillary log files for each of the other tasks. If you re-execute a particular run that intends to write to logs that have the same name as these ancillary files, the run will probably hang. You must dispose of these ancillary log files first.

David Wong, Nov 2011

If you compile CMAQ and encounter the following error messages:

/share/linux86_64/wdx/lib/x86_64i/ifc/mpich/lib/libmpich.a(ibv_param.o): In function `rdma_get_control_parameters': ibv_param.c:(.text+0x457): undefined reference to `ibv_get_device_name' ibv_param.c:(.text+0x470): undefined reference to `ibv_query_device'

it indicates that the system could not locate/include two libraries and you need to include them in the Makefile:

  libibumad.a   (this is InfiniBand related)
  librdmacm.a  (this is for remote direct memory access handling)

Jeff Young, Nov 2011

The ICON preprocessor has a parallel option. This was introduced because it may be possible that you want to generate a huge ICON file that will require too much memory on a single processor. Current processors generally have much more memory available than previously, so it's unlikely you would need to compile and run in parallel. But in case you do, the option is there. To compile you need to set the "ParOpt" option in the "bldit" script. To run, use the run.icon.par script. Its default settings are for 4 processor, which is probably sufficient. You may need to modify the script for the particular parallel platform you intend to use.