Slurm Fairshare
Computational resources on the supercomputer are free for ASU faculty, students, and collaborators. To keep availability equitable, submitted jobs in the queue are prioritized based on recent usage via the submitting user's Fairshare score, which scales a base priority by a factor dependent on recent usage.
Understanding Fairshare and Equitable Use
A fairshare score ranges from 0.0 to 1.0. A user's fairshare halves for every 10,000 core-hour equivalents (CHE) of usage, e.g., 20,000 CHE consumed today would reduce a user's fairshare to 0.25. More specifically,
FairShare = 2^{-current_core_hour_equivalent_usage / 10,000}.
Usage is "forgotten" at an exponential rate, with a half-life of one week. For instance 20,000 CHE consumed today would be "remembered" as 10,000 a week from now, and after 26 weeks, as 1.07 core-second equivalents. For more details on the way fairshare is dynamically controlled on RC's systems, see this 2020 PEARC proceedings paper.
CHE are tracked based on a linear combination of different hardware allocations, i.e.,
CHE = (core-hour equivalents) = (
(number of cores)
+ (total RAM allocated) / (4 GiB)
+ (GPU weight) * (number of GPUs)
) * (job runtime)
The GPU weight depends on the GPU type and the supercomputer:
| Sol GPU | Weight | Phoenix GPU | Weight |
|---|---|---|---|
A100 MIG slice (a100.20gb) | 18 | V100 | 30 |
| A30, A100, L40S, MI200 | 28 | A100 | 40 |
| H100 | 43 | A100 MIG slice (3g.20gb) | 35 |
| H200 | 48 | Other GPU types | 31 |
| GH200 | 63 | ||
| Other accelerators (e.g., Gaudi) | 3 |
Thus, on Sol, utilizing a single core with four GiB of RAM and one A100 GPU for four hours would equate to (1 + 1 + 28) × 4 = 120 CHE. Researchers who carefully manage their hardware allocations to maximize job efficiency (see our documentation page on seff and mysacct for tools to track efficiency) will experience less impact on their FairShare.
All jobs will eventually run; however, researchers with higher recent utilization (resulting in a lower score) may experience longer wait times as other jobs are prioritized.
Requesting more or fewer resources does not affect your position in line. However, jobs have an opportunity to allocate if your job requests resources that do not interfere with the fixed-schedule of higher-priority jobs already in the queue; this is called back-filling, potentially allowing jobs with lower priorities to run ahead-of-schedule.
Checking Fairshare Score
The fairshare score is printed out in the login welcome text. You can also check it at any time with the myfairshare command.
The value in the last column is the real fairshare score or the final calculation result. Please re-run the command if the output is broken or abnormal, and you may need to re-run it multiple times.
Working with Fairshare Score
Here are two examples of how to work around a low fairshare score and get jobs started as soon as possible:
| # | Scenarios | Consequence | Workaround |
|---|---|---|---|
| 1 | A job asking for 300 CPUs and 7 days. | This is a very bulky job and will take a very long waiting time even with a perfect fairshare score, given how busy the supercomputers are. | Break the job into many small jobs (for example, 1 CPU and 1 hour each) using the htc partition, which fit into gaps in the schedule much more easily. Alternatively, -p general -q private lets jobs run on idle privately owned nodes, at the risk of preemption. |
| 2 | Launching thousands of small jobs one after another using a Python script. | Fairshare is charged for the resources each job actually uses, however it was submitted. As the early jobs run and use resources, the fairshare score drops and the later jobs wait longer. Submitting in a tight loop can also hit the per-user submission limits. | Submit the work as a job array instead: one submission, all tasks queued at once, and much less load on the scheduler. Arrays do not reduce the fairshare charge, which is the same as for individual jobs. A single array can have at most 50,000 tasks on Sol. |
Commands to Check Slurm Accounts
Check Fairshare Across All Accounts
$ myfairshare
Account User RawUsage_CHE RawFairShare TargetFairShare RealFairShare
grp_mylab1 rcsparky 3.4 0.619309 0.9997664 0.6193090
class_asu101sp+ rcsparky 20.6 0.061676 0.9985728 0.0616760
The user rcsparky has two accounts with usage on both. The RawUsage_CHE shows the number of Core-Hour-Equivalents run under this account that are used to calculate fairshare, and the RealFairShare column shows the fairshare value for this account.
List Your Slurm Accounts and Associations and QoS:
$ myaccounts
User Def Acct Account QOS
---------- ---------------- ---------------- ----------------------------------
rcsparky grp_mylab1 grp_mylab1 debug,private,public
rcsparky grp_mylab1 class_asu101spr+ class
The user rcsparky belongs to two accounts:
-
Research group account:
grp_mylab1- This account is the default account (Def Acct) and will be used for job submissions unless the
-Aflag specifies a different account. - It supports the QoS levels:
debug,private, andpublic.
- This account is the default account (Def Acct) and will be used for job submissions unless the
-
Class account:
class_asu101spring2025- This account allows submitting jobs only to the class QoS.
If the information in myaccounts is cut off, as indicated by the + symbol in the output, use the -p flag:
myaccounts -p
Submitting Jobs To Different Slurm Accounts
To target a specific account for a job, you can specify the -A flag with the account name. If -A is not specified, the job will be submitted to your default account. Check which account is your default with sacctmgr show user $USER format=user,defaultaccount.
- sbatch-research
- sbatch-class
- interactive-research
- interactive-class
#SBATCH -A grp_mylab
#SBATCH -A class_asu101spring2025
salloc -A grp_mylab
salloc -A class_asu101spring2025