Skip to main content

Partitions and QoS

Every job runs in a partition and under a QoS (Quality of Service).

  • The partition (-p) decides which group of nodes your job can run on: Research Computing-owned nodes, privately owned (lab) nodes, or nodes with special hardware.
  • The QoS (-q) decides how long your job may run, whether it can be preempted, and any per-user limits.

If you specify neither, your job goes to the htc partition with the public QoS, with a four-hour limit. For most work, public (up to 7 days) or htc (up to 4 hours) is the right choice.

At a glance​

PartitionMax timeWhat it is forQoS accepted
htc (default)4 hoursShort CPU and GPU jobs, on Research Computing and privately owned nodes; never preemptedpublic, debug
public7 days (14 with long)General CPU and GPU work on Research Computing-owned nodespublic, long, debug, class
general14 days (30 with grp_)Privately owned nodes: owners, and preemptible use by everyone elsegrp_<labname>, private, debug
lightwork1 dayLight interactive work: editors, compiling, environmentspublic
highmem7 days (14 with long)Jobs needing more memory than a standard node (up to 2 TB)public, private, long, debug, some grp_
gaudi7 daysIntel Gaudi 2 acceleratorspublic, class_gaudi
arm7 daysNVIDIA Grace Hopper (GH200), aarch64public, private, debug, some grp_
fpga7 daysFPGAs, NEC Vector Engine, AMD MI210; one user per nodepublic

Partitions​

htc​

The High-Throughput Computing (htc) partition is the default, and it is the best place for any job that can finish within four hours. It spans Research Computing-owned nodes and idle privately owned nodes, so short jobs usually start soonest here. By arrangement with the node owners, htc jobs submitted with the public QoS are never preempted. It accepts CPU and GPU jobs.

#SBATCH -p htc -q public -t 0-4

public​

The public partition contains all Research Computing-owned nodes, both CPU and GPU, and allows jobs of up to 7 days. Use it with the public, long, debug or (on Sol) class QoS.

#SBATCH -p public -q public -t 7-0

general​

The general partition contains all privately owned (lab-purchased) nodes. There are two ways to use it:

  • Members of the owning lab submit with their lab's grp_<labname> QoS. Their jobs may run for up to 30 days and start ahead of all other jobs on their lab's nodes.
  • Everyone else may use idle lab nodes with the private QoS. These jobs can be preempted (cancelled) when the owners need their hardware. Without preemption the limit is 14 days on Sol and 7 days on Phoenix.
#SBATCH -p general
#SBATCH -q grp_<labname>
#SBATCH -t 30-0

highmem​

The highmem partition is for jobs that need more memory than a standard compute node provides (on Sol, standard nodes have 512 GiB and highmem nodes have 1 to 2.2 TiB; on Phoenix, most standard nodes have 128 GiB and highmem nodes have about 1.5 TiB). Compute speed is about the same as standard nodes. Jobs may run for up to 7 days. On Sol, the long QoS extends this to 14 days. Reach out to Research Computing staff if you need it, and we can help check that your job is a good fit for these nodes.

#SBATCH -p highmem -q public -t 7-0

lightwork​

The lightwork partition is for jobs that need relatively little computing power or sit idle much of the time, for example creating mamba environments, compiling software, VSCode tunnels, or bulk file operations. The maximum job time is 24 hours, and jobs should use no more than 8 CPU cores.

#SBATCH -p lightwork -q public -t 0-24
info

Jobs that keep their cores fully busy belong in htc, public or general, where cores are not shared. Jobs that run cores at over 99% for long periods, or that request so much that others cannot use the partition, are subject to cancellation. Repeated misuse will make you ineligible for lightwork.

gaudi (Sol only)​

The gaudi partition contains nodes with eight Intel Gaudi 2 (HL-225) accelerators each. Request them with --gres=gpu:hl225:<count>. Jobs may run for up to 7 days with the public QoS; courses using Gaudi hardware use the class_gaudi QoS. See requesting Gaudi accelerators.

arm (Sol only)​

The arm partition is for aarch64 (ARM) workloads. On Sol, every node in it is an NVIDIA Grace Hopper (GH200). ARM software is not interchangeable with x86-64 software, so check that your software supports ARM before using this partition. See requesting Grace Hopper allocations.

fpga (Sol only)​

The fpga partition holds specialized accelerator nodes. Each node is allocated to one user at a time, so request the node that has the hardware you need rather than taking whatever is free:

NodeCPUAccelerator
sfpga01aAMD EPYCAMD Xilinx FPGA
sfpga01iIntel XeonBittWare (Intel/Altera) FPGA
sfpga01nAMD EPYCNEC Vector Engine
sfpga01uAMD EPYCnone currently installed
sg250AMD EPYC2× AMD Instinct MI210 GPU (--gres=gpu:mi200:<count>)

See using FPGAs.

QoS​

public​

The public QoS is the standard choice for every partition except general. If you do not specify a QoS, public is used.

#SBATCH -p public -q public -t 7-0

debug​

debug is for quickly testing job scripts before a full run: catching syntax errors, wrong paths, or missing modules with a small dataset. Jobs are limited to 15 minutes, and each user may have one debug job running (and two queued) at a time. In exchange, they usually start quickly.

On Sol, debug works in the public, general, htc, highmem and arm partitions. On Phoenix, it works in public.

#SBATCH -p public -q debug -t 15

private​

The private QoS lets you run on idle privately owned nodes in the general partition for longer than the four-hour htc window. In return, your job is preemptible: it is cancelled if the owning lab needs the node. Depending on overall load, this can shorten your wait considerably. It suits work that checkpoints regularly, especially on privately owned GPUs.

  • Maximum time: 14 days on Sol, 7 days on Phoenix.
  • Sol per-user limit: 161,280 GPU running minutes (for example, 16 GPUs for 7 days) across all your private jobs. Further private jobs wait until running ones finish, giving other users a turn.
#SBATCH -p general
#SBATCH -q private

grp_labname​

Each lab that purchases hardware receives a group QoS named grp_<labname>. Members of the lab use it to run on their lab's nodes:

  • Jobs may run for up to 30 days.
  • Use of your own hardware has practically no effect on your fairshare.
  • Your jobs start ahead of every other job waiting for your lab's nodes, regardless of fairshare.
  • Your jobs can preempt private jobs on your nodes, but not htc jobs (see Preemption).
#SBATCH -p general
#SBATCH -q grp_<labname>
#SBATCH -t 30-0

long​

The long QoS extends jobs on Research Computing-owned hardware from 7 days to 14 days. It works with sbatch only, not interactive sessions, in the public partition (and highmem on Sol).

#SBATCH -p public
#SBATCH -q long
#SBATCH -t 14-0

long is not available by default; it is granted case by case. If you are interested, be ready to share a job ID that shows the need and makes effective use of the cores it requests. Feel free to ask staff, and see the Slurm efficiency reporter.

class (Sol only)​

The class QoS is for users who have access to Sol through an academic course. Its limits help course jobs start sooner and share resources fairly:

  • Per job: at most 32 CPU cores, 320 GB memory and 4 GPUs, for up to 24 hours.
  • Per user:
    • at most 2 running jobs and 10 queued jobs
    • 960 GPU running minutes, shared across running jobs (for example, 1 GPU for 16 hours or 4 GPUs for 4 hours)
#SBATCH -p public
#SBATCH -q class
#SBATCH -t 1-0

If you have both a course account and a research account, choose the account for each job with -A:

#SBATCH -p public
#SBATCH -q class
#SBATCH -t 1-0
#SBATCH -A class_asus101spring2025

To see which accounts you have, run myfairshare or myaccounts:

$ myfairshare
Account User RawUsage_CHE RawFairShare TargetFairShare RealFairShare
class_asu101spring2025 rcsparky 0.0 1.000000 1.0000000 1.0000000
grp_mygroup rcsparky 61.1 0.043506 0.9957724 0.0435060

$ myaccounts
User Def Acct Account QOS
---------- ---------------- ---------------- --------------------------------
rcsparky grp_rcadmins class_asu101s2025 class
rcsparky grp_mylab grp_mylab public,private,debug

class_gaudi (Sol only)​

class_gaudi is the course QoS for the gaudi partition. Jobs may run for up to 24 hours.

#SBATCH -p gaudi
#SBATCH -q class_gaudi
#SBATCH -t 1-0

Preemption on privately owned nodes​

Privately owned nodes are shared with the community when the owning lab is not using them. The rules below keep that hardware productive while guaranteeing the lab access to what it bought.

Owners come first. A grp_<labname> job always starts ahead of every other job waiting for that lab's nodes, regardless of fairshare. Whether it can start immediately depends on what is already running there:

Job already running on the lab's nodeWhat happens when the owner needs the node
htc partition, public QoS (at most 4 hours)Never preempted. It finishes normally; the owner's job starts next.
private QoSPreempted: cancelled so the owner's job can start, once it has run for at least 30 seconds.

When a private job is preempted:

  • It is cancelled, not requeued: Slurm does not restart it automatically. Resubmit it, ideally from a checkpoint.
  • sacct shows its state as PREEMPTED.
  • A job is preempted only when an owner's job cannot otherwise start on that node.
tip

If your work can be split into pieces of four hours or less, htc gives you access to the same privately owned hardware with no risk of preemption. For longer private jobs, write checkpoints often so that a preemption costs minutes rather than days.

htc jobs can also start on a lab-owned node through backfilling, but only when doing so does not delay a higher-priority job, and owners' jobs are always higher priority.