Skip to main content

Snakemake Basics

Snakemake is a workflow management tool that lets you define analysis pipelines using a Python-based language. On ASU Supercomputers, Snakemake submits each rule as a sbatch job to the scheduler.

The key idea is to run your main Snakemake process on minimal resources with a long walltime, while the child jobs it spawns request only what they need for short durations.

Main Snakemake Job​

Submit your Snakemake controller as a lightweight, long-running job. It doesn't do heavy computation — it just monitors and submits child jobs.

#!/bin/bash
#SBATCH --job-name=snakemake_main
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=7-00:00:00
#SBATCH --partition=public
#SBATCH --qos=public

module load mamba/latest
source activate snakemake # your own environment, see the note below

snakemake --executor slurm --jobs 50 \
--default-resources slurm_partition=htc slurm_account=<your_account> \
runtime=120 mem_mb=4000 slurm_extra="'--qos=public'"
note

Snakemake 8 and newer removed the --cluster "sbatch ..." option. Jobs are submitted to Slurm through the --executor slurm option, which needs the separate snakemake-executor-plugin-slurm package. The plugin is not included in the central Snakemake installs, so create your own environment once (in an interactive session) that contains both:

module load mamba/latest
mamba create -n snakemake -c conda-forge -c bioconda snakemake snakemake-executor-plugin-slurm

If you already have a Snakemake environment, add the plugin after activating it with pip install snakemake-executor-plugin-slurm.

Use the htc partition for child jobs up to 4 hours and the public partition for longer jobs (up to 7 days). Replace <your_account> with your Slurm account name.

Child Job Resources​

Define resources per rule in your Snakefile so each child job is short and lean. This helps your jobs pass through the scheduler faster and works in your favor for fairshare scheduling.

rule align:
input: "reads/{sample}.fastq"
output: "aligned/{sample}.bam"
threads: 4
resources:
mem_mb=8000,
runtime=120 # minutes
shell:
"bwa mem -t {threads} ref.fa {input} | samtools sort -o {output}"

Cluster Profile​

For cleaner configuration, create a Snakemake profile under ~/.config/snakemake/<profile_name>/config.yaml to set the executor, default and per-rule resource values instead of putting everything on the command line, then run snakemake --profile <profile_name>.

Tips​

  • Keep child jobs short. Shorter walltimes improve queue priority under fairshare and start sooner.
  • Request only the resources you need. Over-requesting memory or CPUs wastes your fairshare allocation.
  • Use --rerun-incomplete if your main job times out so Snakemake picks up where it left off.
  • Snakemake is considered an advanced use case, please contact the Research Computing team for support if needed.