Skip to main content

Slurm Fairshare

Computational resources on the supercomputer are free for ASU faculty, students, and collaborators. To keep availability equitable, submitted jobs in the queue are prioritized based on recent usage via the submitting user's Fairshare score, which scales a base priority by a factor dependent on recent usage.

Understanding Fairshare and Equitable Use​

A fairshare score ranges from 0.0 to 1.0. A user's fairshare halves for every 10,000 core-hour equivalents (CHE) of usage, e.g., 20,000 CHE consumed today would reduce a user's fairshare to 0.25. More specifically,

FairShare = 2^{-current_core_hour_equivalent_usage / 10,000}.

Usage is "forgotten" at an exponential rate, with a half-life of one week. For instance 20,000 CHE consumed today would be "remembered" as 10,000 a week from now, and after 26 weeks, as 1.07 core-second equivalents. For more details on the way fairshare is dynamically controlled on RC's systems, see this 2020 PEARC proceedings paper.

CHE are tracked based on a linear combination of different hardware allocations, i.e.,

CHE = (core-hour equivalents) = (
    (number of cores)
    + (total RAM allocated) / (4 GiB)
    + 3 * (number of Multi-Instance GPU slices)
    + 20 * (number of A30 GPUs)
    + 25 * (number of A100 GPUs)
    + 40 * (number of H100 GPUs)
) * (job runtime)

Thus, utilizing a single core with four GiB of RAM and one A100 GPU for four hours would equate to approximately 108 CHE. Researchers who carefully manage their hardware allocations to maximize job efficiency (see our documentation page on seff and mysacct for tools to track efficiency) will experience less impact on their FairShare.

info

All jobs will eventually run; however, researchers with higher recent utilization (resulting in a lower score) may experience longer wait times as other jobs are prioritized.

tip

Requesting more or fewer resources does not affect your position in line. However, jobs have an opportunity to allocate if your job requests resources that do not interfere with the fixed-schedule of higher-priority jobs already in the queue; this is called back-filling, potentially allowing jobs with lower priorities to run ahead-of-schedule.

Checking Fairshare Score​

The fairshare score is printed out in the login welcome text. And there are two commands available check the fairshare score (they are equivalent):

  • myfairshare
  • mybalance

The value in the last column is the real fairshare score or the final calculation result. Please re-run these commands if the output is broken or abnormal, and you may need to re-run them multiple times.

Working with Fairshare Score​

Here are two examples of how to work around a low fairshare score and get jobs started as soon as possible:

#ScenariosConsequenceWorkaround
1A job asking for 300 CPUs and 7 days.This is a very bulky job and will take a very long waiting time even with a perfect fairshare score, given how busy the supercomputers are.Break the job into 300 * 7 * 24 = 50400 small jobs asking for 1 CPU and 1 hour per job, using the htc partition. Using the private QOS also helps.
2Launching 50400 small jobs one after one using a python script.Each submission will take a deduction on the fairshare score, so the waiting time will get longer and longer, some of the jobs will wait for days.Instead of submitting the jobs one by one, launching a job array with 50400 sub-jobs will one take a one-time deduction on the fairshare score. So all the sub-jobs can start queuing using the same fairshare score.

Commands to Check Slurm Accounts​

Check Fairshare Across All Accounts​

$ myfairshare 
Account         User      RawUsage_CHE  RawFairShare  TargetFairShare  RealFairShare
grp_mylab1      jeburks2  3.4           0.619309      0.9997664        0.6193090
class_asu101sp+ jeburks2  20.6          0.061676      0.9985728        0.0616760

Thie user jeburks2 has two accounts with usage on both. The RawUsage_CHE shows the number of Core-Hour-Equivalents run under this account that are used to calculate fairshare, and the RealFairShare column shows the fairshare value for this account. For more information on Fairshare, see Slurm Fairshare Score.

List Your Slurm Accounts and Associations and QoS:​

$ myaccounts
      User       Def Acct        Account                                QOS
---------- ---------------- ---------------- ----------------------------------
  jeburks2      grp_mylab1      grp_mylab1           debug,private,public
  jeburks2      grp_mylab1 class_asu101spr+                          class

The user jeburks2 belongs to two accounts:

  1. Research group account: grp_mylab1

    • This account is the default account (Def Acct) and will be used for job submissions unless the -A flag specifies a different account.
    • It supports the QoS levels: debug, private, and public.

  2. Class account: class_asu101spring2025

    • This account allows submitting jobs only to the class QoS.
tip

If the information in myaccounts is cut off, as indicated by the + symbol in the output, use the -p flag:

myaccounts -p

Submitting Jobs To Different Slurm Accounts​

To target a specific account for a job, you can specify the -A flag with the account name. If -A is not specified, the job will be submitted to your default account, which is the first account you joined.

#SBATCH -A grp_mylab