Skip to main content

Requesting Resources on the Supercomputer

CPUs and memory​

CPUs and memory can be requested independently. If you do not specify memory, each CPU core comes with 2 GB. Jobs that request GPUs get a larger default; see Memory for GPU jobs.

salloc and interactive are interchangeable.

CommandResources
salloc -c 11 CPU, 2 GB
salloc -c 6464 CPUs, 128 GB shared between them
salloc -c 32 --mem=80GB32 CPUs, 80 GB shared between them
salloc -c 128 --mem=0128 CPUs, all of the node's allocatable memory
salloc -c 16 --mem-per-cpu=4G16 CPUs, 4 GB dedicated to each, 64 GB total

Memory can be requested as one total (--mem) or per CPU (--mem-per-cpu). In most cases --mem is the better choice, unless you are working specifically with OpenMP or multithreading.

info

Every node keeps a small amount of memory for the operating system: 3.5 GB on Sol, and 2–3.5 GB on Phoenix. The memory you can request on a node, including what --mem=0 gives you, is the node's physical memory minus this reserve. A request for exactly a node's advertised size (for example --mem=512G) will not fit; ask for a little less, or use --mem=0.

Requesting multiple nodes​

MPI (Message Passing Interface) spreads one job across several physical nodes, each with its own memory. Not all software supports MPI; it must be built for it.

warning

-N only helps MPI jobs. If your software is not MPI-enabled, requesting more nodes will not make it faster.

  • -N is the number of nodes.
  • -n is the number of tasks.
  • -c is the number of CPU cores per task.

MPI processes bind to tasks, so as a rule of thumb MPI jobs request tasks (-n), serial and multithreaded jobs request cores (-c), and hybrid jobs request both.

#SBATCH -N 2 # two nodes
#SBATCH -n 64 # 64 MPI tasks, spread across both nodes
#SBATCH -c 1 # 1 core per task

Multi-node MPI jobs need a fast interconnect between nodes; you can require one with the ib or opa node feature. See the official Slurm documentation for more: Slurm Workload Manager - sbatch.

GPU Accelerators​

Request GPUs with -G (or --gres=gpu:). On its own, -G 1 gives you the first available GPU of any type. Add a type, as in -G a100:1, to get a specific one. GPUs exist in several partitions, so choose the partition that matches your needs.

Memory for GPU jobs​

If a GPU job does not specify memory, it gets whichever is larger: 2 GB per CPU core, or a per-GPU minimum of 24 GB per GPU on Sol and 6 GB per GPU on Phoenix. For example, on Sol, -c 4 -G 1 gets 24 GB, and -c 16 -G 1 gets 32 GB. A notice on your job's error output reports the default that was applied.

An explicit --mem, --mem-per-cpu, or --mem-per-gpu is always honored, even if it is below the minimum; you will see a notice in that case. --mem=0 gives you all of the node's allocatable memory, as for CPU jobs.

#SBATCH -p htc
#SBATCH -q public
#SBATCH -t 0-4
#SBATCH -G 1

GPU types​

Use these names with -G <type>:<count>:

-G typeGPUPartitions
a100NVIDIA A100, 80 GB (the most common GPU on Sol)public, htc, general
a100.40gbNVIDIA A100, 40 GBgeneral, htc
a100.20gb20 GB slice of an A100public, htc, lightwork
a30NVIDIA A30public, htc, general
l40NVIDIA L40Sgeneral, htc
h100NVIDIA H100general, htc
h200, h200.71gb, h200.35gbNVIDIA H200, whole GPU or a 71 GB / 35 GB slicegeneral, htc
gh200NVIDIA Grace Hopperarm
hl225Intel Gaudi 2gaudi
mi200AMD Instinct MI210fpga

Grace Hopper/GH200​

Grace Hopper (GH200) nodes are in the arm partition. A GH200 combines an H100-class GPU with an ARM (Grace) CPU on one module. That brings some caveats, notably that software must be compiled for aarch64. Read more on the Grace Hopper page.

#SBATCH -p arm
#SBATCH -q public
#SBATCH -t 0-4
#SBATCH -G 1

Gaudi Accelerators​

To use the Intel Gaudi 2 AI accelerators, submit to the gaudi partition.

Request accelerators with -G <count> or --gres=gpu:hl225:<count>; each node has eight. Each node has 152 cores, so take 18 cores per accelerator (-c 18 for one).

#SBATCH -p gaudi
#SBATCH -q public
#SBATCH -N 1
#SBATCH -G 1
#SBATCH -t 0-4
#SBATCH -c 18

Gaudi accelerators need their own Python environment to reach the hardware. In Jupyter, use the tested kernels named "gaudi-pytorch" and "gaudi-pytorch-diffusion*"; other kernels lack the packages and configuration that Gaudi needs.

FPGA​

FPGAs (Field-Programmable Gate Arrays) are reconfigurable chips that you program to implement custom digital logic. Unlike CPUs or GPUs, which run a fixed instruction set, an FPGA lets you define the hardware's behavior directly.

Sol provides FPGA platforms from both AMD Xilinx and Intel, as well as the NEC Vector Engine. Each is on its own node in the fpga partition, so request it by node name:

salloc -w sfpga01a --exclusive -p fpga --mem=0

OR

#SBATCH -w sfpga01a
#SBATCH -p fpga
#SBATCH --mem=0
#SBATCH --exclusive

Node features (--constraint)​

Every node carries feature tags that describe its hardware. Use --constraint (-C) to require them, for example a particular CPU generation, an InfiniBand connection, or a GPU with a given amount of memory. As of October 2026, the tags describe the hardware actually installed in each node.

#SBATCH -C zen4 # AMD EPYC Zen 4
CategoryTagsNotes
CPU vendorintel, epycepyc = AMD EPYC
AMD CPU generationnaples, zen2, zen3, zen4, zen5Sol: zen3, zen4, zen5. Phoenix: naples, zen2, zen3
Intel CPU generationbroadwell, skylake, cascadelake, icelake, sapphirerapids, emeraldrapidsPhoenix is mostly broadwell and cascadelake. Sol has a few Intel nodes (cascadelake, icelake, sapphirerapids, emeraldrapids)
Instruction setavx512CPUs with AVX-512
CPU architecturearmaarch64 nodes (Sol Grace Hopper); everything else is x86-64
Interconnectib, opaib = InfiniBand, opa = Omni-Path (Phoenix only). Nodes with neither have Ethernet only and suit single-node jobs
GPU modela100, a30, l40s, h100, h100nvl, h200, gracehopper (Sol); v100, a100, a40, a6000, a4000, rtx6000, rtx5000, rtx4500, rtx2080, gtx1080 (Phoenix)rtx2080 covers both the RTX 2080 and 2080 Ti
GPU memorya100_20, a100_40, a100_80, h100_80, v100_16, v100_32Memory (GB) of each GPU or GPU slice on the node
Other acceleratorsgaudi, hl225, fpga, xilinx, bittware, vector, mi200See Gaudi and FPGA
MemoryhighmemLarge-memory nodes
tip

-G chooses the GPU type and count; -C filters which nodes are eligible. A constraint that no node in your partition satisfies leaves your job pending or rejected with Invalid feature specification. The phi tag no longer exists on any cluster.

The Phoenix supercomputer mixes several Intel CPU micro-architectures, such as Cascade Lake and Broadwell. Sol is mostly AMD EPYC (zen3 and zen4) with a few Intel nodes. Software can run differently, or fail with "illegal instruction", on a CPU generation older than the one it was compiled for. If you build with CPU-specific optimizations, constrain your jobs to that generation.