MPI Launchers
Jobs that scale massively parallel and split into independent tasks rely on MPI modules and launchers to orchestrate.
The most common MPI module is Intel MPI, available through oneapi modules. Besides oneapi, OpenMPI, MPICH and others exist.
Usage of each of the modules/launchers is split by supercomputer, then by module.
Sol MPI
- Intel oneAPI
- Intel Parallel Studio
- OpenMPI
- MPICH
module load intel/intel-oneapi-2022.1.0
mpiexec.hydra -genvall -np $SLURM_NTASKS "./my-simulation"
module load intel/parallel-studio-2020.4
mpiexec.hydra -genvall -np $SLURM_NTASKS "./my-simulation"
Example script available at: /packages/public/sol-sbatch-templates/templates/example-mpi-with-intel-job/main.sh
module load openmpi/4.1.5
srun --export=ALL --mpi=pmix "./my-simulation"
Example script available at: /packages/public/sol-sbatch-templates/templates/example-mpi-with-hpcx-job/main.sh
module load mpich/4.2.2
mpirun -launcher slurm "./my-simulation"
Example script available at: /packages/public/sol-sbatch-templates/templates/example-mpi-with-mpich-job/main.sh
Phoenix MPI
- Intel oneAPI
- OpenMPI 4
- OpenMPI 5
- MPICH
module load intel/oneapi
mpiexec.hydra -genvall -np $SLURM_NTASKS "./my-simulation"
Example script available at: /packages/public/phx-sbatch-templates/templates/example-mpi-with-intel-job/main.sh
Note: There have been a few cases where users have needed to add the following to their job scripts:
export HYDRA_LAUNCHER_EXTRA_ARGS="--export=ALL"
Jobs with more than one MPI rank, including single-node jobs: Phoenix nodes are on two different networks (Mellanox InfiniBand and Omni-Path). Set FI_PROVIDER=verbs only on the InfiniBand nodes. Setting it on Omni-Path nodes crashes the job, and those nodes work with the default. Without it, jobs on the InfiniBand nodes fail at startup with OFI get address vector map failed. This is the supported configuration going forward; the Phoenix node image will not change to make the default work on the InfiniBand nodes.
if [ -e /sys/class/infiniband/mlx5_0 ]; then
export FI_PROVIDER=verbs
fi
This checks the first node of the job, so all of a job's nodes must be on the same network; you can request one with #SBATCH --constraint=ib or #SBATCH --constraint=opa.
Single-node jobs only:
export FI_PROVIDER=shm
module load openmpi/4.1.5
srun --export=ALL --mpi=pmix "./my-simulation"
Example script available at: /packages/public/phx-sbatch-templates/templates/example-mpi-with-hpcx-job/main.sh
module load openmpi/5.0.6
mpirun -np $SLURM_NTASKS "./my-simulation"
Example script available at: /packages/public/phx-sbatch-templates/templates/example-mpi-with-openmpi5-job/main.sh
module load mpich/4.1.2
mpirun -launcher slurm "./my-simulation"
Example script available at: /packages/public/phx-sbatch-templates/templates/example-mpi-with-mpich-job/main.sh