Requesting Resources on the Supercomputer
CPUs and memory
CPUs and memory can be requested independently. If you do not specify memory, each CPU core comes with 2 GB. Jobs that request GPUs get a larger default; see Memory for GPU jobs.
- interactively
- sbatch
salloc and interactive are interchangeable.
| Command | Resources |
|---|---|
salloc -c 1 | 1 CPU, 2 GB |
salloc -c 64 | 64 CPUs, 128 GB shared between them |
salloc -c 32 --mem=80GB | 32 CPUs, 80 GB shared between them |
salloc -c 128 --mem=0 | 128 CPUs, all of the node's allocatable memory |
salloc -c 16 --mem-per-cpu=4G | 16 CPUs, 4 GB dedicated to each, 64 GB total |
#SBATCH directives take the same values as salloc or interactive:
#SBATCH -c 1 # 1 CPU core
#SBATCH -c 32 # 32 CPU cores
#SBATCH --mem=80GB # 80 GB shared between all CPUs
#SBATCH -c 128 # 128 CPU cores
#SBATCH --mem=0 # all of the node's allocatable memory
#SBATCH -c 16 # 16 CPU cores
#SBATCH --mem-per-cpu=4G # 4 GB per CPU, 64 GB total
Memory can be requested as one total (--mem) or per CPU (--mem-per-cpu). In most cases --mem is the better choice, unless you are working specifically with OpenMP or multithreading.
Every node keeps a small amount of memory for the operating system: 3.5 GB on Sol, and 2–3.5 GB on Phoenix. The memory you can request on a node, including what --mem=0 gives you, is the node's physical memory minus this reserve. A request for exactly a node's advertised size (for example --mem=512G) will not fit; ask for a little less, or use --mem=0.
Requesting multiple nodes
MPI (Message Passing Interface) spreads one job across several physical nodes, each with its own memory. Not all software supports MPI; it must be built for it.
-N only helps MPI jobs. If your software is not MPI-enabled, requesting more nodes will not make it faster.
-Nis the number of nodes.-nis the number of tasks.-cis the number of CPU cores per task.
MPI processes bind to tasks, so as a rule of thumb MPI jobs request tasks (-n), serial and multithreaded jobs request cores (-c), and hybrid jobs request both.
- MPI
- hybrid MPI + threads
#SBATCH -N 2 # two nodes
#SBATCH -n 64 # 64 MPI tasks, spread across both nodes
#SBATCH -c 1 # 1 core per task
#SBATCH -N 2 # two nodes
#SBATCH -n 8 # 8 MPI tasks
#SBATCH -c 8 # 8 threads (cores) per task, 64 cores total
Multi-node MPI jobs need a fast interconnect between nodes; you can require one with the ib or opa node feature. See the official Slurm documentation for more: Slurm Workload Manager - sbatch.
GPU Accelerators
Request GPUs with -G (or --gres=gpu:). On its own, -G 1 gives you the first available GPU of any type. Add a type, as in -G a100:1, to get a specific one. GPUs exist in several partitions, so choose the partition that matches your needs.
Memory for GPU jobs
If a GPU job does not specify memory, it gets whichever is larger: 2 GB per CPU core, or a per-GPU minimum of 24 GB per GPU on Sol and 6 GB per GPU on Phoenix. For example, on Sol, -c 4 -G 1 gets 24 GB, and -c 16 -G 1 gets 32 GB. A notice on your job's error output reports the default that was applied.
An explicit --mem, --mem-per-cpu, or --mem-per-gpu is always honored, even if it is below the minimum; you will see a notice in that case. --mem=0 gives you all of the node's allocatable memory, as for CPU jobs.
- first available (htc, sbatch)
- interactive
- interactive, 1× A100
- sbatch, 4× A100
- preemptible, 2× A30
- AMD MI210
#SBATCH -p htc
#SBATCH -q public
#SBATCH -t 0-4
#SBATCH -G 1
salloc -G 1
salloc -G a100:1
#SBATCH -G a100:4
#SBATCH -p general -q private -G a30:2
salloc -p fpga -G mi200:1
OR
#SBATCH -p fpga
#SBATCH -G mi200:1
These are AMD Instinct MI200-series accelerators (MI210). They are not FPGAs, but they are not CUDA-enabled either, so for convenience they are kept in the fpga partition.
GPU types
Use these names with -G <type>:<count>:
- Sol
- Phoenix
-G type | GPU | Partitions |
|---|---|---|
a100 | NVIDIA A100, 80 GB (the most common GPU on Sol) | public, htc, general |
a100.40gb | NVIDIA A100, 40 GB | general, htc |
a100.20gb | 20 GB slice of an A100 | public, htc, lightwork |
a30 | NVIDIA A30 | public, htc, general |
l40 | NVIDIA L40S | general, htc |
h100 | NVIDIA H100 | general, htc |
h200, h200.71gb, h200.35gb | NVIDIA H200, whole GPU or a 71 GB / 35 GB slice | general, htc |
gh200 | NVIDIA Grace Hopper | arm |
hl225 | Intel Gaudi 2 | gaudi |
mi200 | AMD Instinct MI210 | fpga |
-G type | GPU | Partitions |
|---|---|---|
v100 | NVIDIA V100 | public, htc, general |
a100 | NVIDIA A100 | general, htc |
3g.20gb | 20 GB slice of an A100 | general, htc |
a40 | NVIDIA A40 | general, htc |
a6000 | NVIDIA RTX A6000 | general, htc |
a4000 | NVIDIA RTX A4000 | general, htc |
rtx6000 | NVIDIA Quadro RTX 6000 | general, htc |
rtx5000 | NVIDIA RTX 5000 Ada | general, htc |
rtx4500 | NVIDIA RTX 4500 Ada | general, htc |
rtx2080 | NVIDIA GeForce RTX 2080 and 2080 Ti | general, htc |
gtx1080 | NVIDIA GeForce GTX 1080 Ti | general, htc |
As of October 2026, the RTX 2080 Ti GPUs that were requested as gtx2080 are now rtx2080, like the other RTX 2080 cards, and the k40 type has been retired. Jobs that still ask for gtx2080 or k40 will be rejected.
Grace Hopper/GH200
Grace Hopper (GH200) nodes are in the arm partition. A GH200 combines an H100-class GPU with an ARM (Grace) CPU on one module. That brings some caveats, notably that software must be compiled for aarch64. Read more on the Grace Hopper page.
- sbatch
- interactive
- interactive (privately owned)
#SBATCH -p arm
#SBATCH -q public
#SBATCH -t 0-4
#SBATCH -G 1
salloc -p arm -G 1
Privately owned Grace Hopper nodes can be used by anyone with the private QoS, for up to 7 days. These jobs can be preempted (cancelled) if a member of the owning lab needs the node.
$ salloc -p arm -G 1 -q private
Gaudi Accelerators
To use the Intel Gaudi 2 AI accelerators, submit to the gaudi partition.
Request accelerators with -G <count> or --gres=gpu:hl225:<count>; each node has eight. Each node has 152 cores, so take 18 cores per accelerator (-c 18 for one).
- sbatch
- interactive
- webportal
- class account sbatch
- class account interactive
- class webportal
#SBATCH -p gaudi
#SBATCH -q public
#SBATCH -N 1
#SBATCH -G 1
#SBATCH -t 0-4
#SBATCH -c 18
salloc -p gaudi -q public -N 1 -G 1 -c 18
salloc -p gaudi -N 1 -G 1 -c 18
#SBATCH -p gaudi
#SBATCH -q class_gaudi
#SBATCH -N 1
#SBATCH -G 1
#SBATCH -t 0-4
#SBATCH -c 18
salloc -p gaudi -q class_gaudi -N 1 -G 1 -c 18
salloc -p gaudi -q class_gaudi -N 1 -G 1 -c 18
Gaudi accelerators need their own Python environment to reach the hardware. In Jupyter, use the tested kernels named "gaudi-pytorch" and "gaudi-pytorch-diffusion*"; other kernels lack the packages and configuration that Gaudi needs.
FPGA
FPGAs (Field-Programmable Gate Arrays) are reconfigurable chips that you program to implement custom digital logic. Unlike CPUs or GPUs, which run a fixed instruction set, an FPGA lets you define the hardware's behavior directly.
Sol provides FPGA platforms from both AMD Xilinx and Intel, as well as the NEC Vector Engine. Each is on its own node in the fpga partition, so request it by node name:
- AMD FPGA
- Intel BittWare
- NEC Vector Engine
salloc -w sfpga01a --exclusive -p fpga --mem=0
OR
#SBATCH -w sfpga01a
#SBATCH -p fpga
#SBATCH --mem=0
#SBATCH --exclusive
salloc -w sfpga01i --exclusive -p fpga --mem=0
OR
#SBATCH -w sfpga01i
#SBATCH -p fpga
#SBATCH --mem=0
#SBATCH --exclusive
salloc -w sfpga01n --exclusive -p fpga --mem=0
OR
#SBATCH -w sfpga01n
#SBATCH -p fpga
#SBATCH --mem=0
#SBATCH --exclusive
Node features (--constraint)
Every node carries feature tags that describe its hardware. Use --constraint (-C) to require them, for example a particular CPU generation, an InfiniBand connection, or a GPU with a given amount of memory. As of October 2026, the tags describe the hardware actually installed in each node.
- CPU generation
- AVX-512
- interconnect
- combined
#SBATCH -C zen4 # AMD EPYC Zen 4
#SBATCH -C avx512 # any CPU with AVX-512 instructions
#SBATCH -C ib # InfiniBand (Sol, some Phoenix nodes)
#SBATCH -C opa # Omni-Path (Phoenix)
#SBATCH -C "cascadelake&opa" # both
#SBATCH -C "zen3|zen4" # either
#SBATCH -G a100:1 -C a100_80 # an A100 with 80 GB
| Category | Tags | Notes |
|---|---|---|
| CPU vendor | intel, epyc | epyc = AMD EPYC |
| AMD CPU generation | naples, zen2, zen3, zen4, zen5 | Sol: zen3, zen4, zen5. Phoenix: naples, zen2, zen3 |
| Intel CPU generation | broadwell, skylake, cascadelake, icelake, sapphirerapids, emeraldrapids | Phoenix is mostly broadwell and cascadelake. Sol has a few Intel nodes (cascadelake, icelake, sapphirerapids, emeraldrapids) |
| Instruction set | avx512 | CPUs with AVX-512 |
| CPU architecture | arm | aarch64 nodes (Sol Grace Hopper); everything else is x86-64 |
| Interconnect | ib, opa | ib = InfiniBand, opa = Omni-Path (Phoenix only). Nodes with neither have Ethernet only and suit single-node jobs |
| GPU model | a100, a30, l40s, h100, h100nvl, h200, gracehopper (Sol); v100, a100, a40, a6000, a4000, rtx6000, rtx5000, rtx4500, rtx2080, gtx1080 (Phoenix) | rtx2080 covers both the RTX 2080 and 2080 Ti |
| GPU memory | a100_20, a100_40, a100_80, h100_80, v100_16, v100_32 | Memory (GB) of each GPU or GPU slice on the node |
| Other accelerators | gaudi, hl225, fpga, xilinx, bittware, vector, mi200 | See Gaudi and FPGA |
| Memory | highmem | Large-memory nodes |
-G chooses the GPU type and count; -C filters which nodes are eligible. A constraint that no node in your partition satisfies leaves your job pending or rejected with Invalid feature specification. The phi tag no longer exists on any cluster.
The Phoenix supercomputer mixes several Intel CPU micro-architectures, such as Cascade Lake and Broadwell. Sol is mostly AMD EPYC (zen3 and zen4) with a few Intel nodes. Software can run differently, or fail with "illegal instruction", on a CPU generation older than the one it was compiled for. If you build with CPU-specific optimizations, constrain your jobs to that generation.