Skip to main content

MPI Launchers

Jobs that scale massively parallel and split into independent tasks rely on MPI modules and launchers to orchestrate.

The most common MPI module is Intel MPI, available through oneapi modules. Besides oneapi, OpenMPI, MPICH and others exist.

Usage of each of the modules/launchers is split by supercomputer, then by module.

Sol MPI​

module load intel/intel-oneapi-2022.1.0

mpiexec.hydra -genvall -np $SLURM_NTASKS "./my-simulation"

Phoenix MPI​

module load intel/oneapi

mpiexec.hydra -genvall -np $SLURM_NTASKS "./my-simulation"

Example script available at: /packages/public/phx-sbatch-templates/templates/example-mpi-with-intel-job/main.sh

info

Note: There have been a few cases where users have needed to add the following to their job scripts:

export HYDRA_LAUNCHER_EXTRA_ARGS="--export=ALL"

Jobs with more than one MPI rank, including single-node jobs: Phoenix nodes are on two different networks (Mellanox InfiniBand and Omni-Path). Set FI_PROVIDER=verbs only on the InfiniBand nodes. Setting it on Omni-Path nodes crashes the job, and those nodes work with the default. Without it, jobs on the InfiniBand nodes fail at startup with OFI get address vector map failed. This is the supported configuration going forward; the Phoenix node image will not change to make the default work on the InfiniBand nodes.

if [ -e /sys/class/infiniband/mlx5_0 ]; then
export FI_PROVIDER=verbs
fi

This checks the first node of the job, so all of a job's nodes must be on the same network; you can request one with #SBATCH --constraint=ib or #SBATCH --constraint=opa.

Single-node jobs only:

export FI_PROVIDER=shm