Snakemake training session

Introduction to Snakemake workflows for beginners

2025-03-25

Past & present contributions:

Emilie Drouineau, I2BC/CEA; Fadwa El-Khaddar, BIOI2 - I2BC/Univ. Paris-Saclay; Chloe Quignot, BIOI2 - I2BC/CEA; Claire Toffano-Nioche, I2BC/CNRS;

Source: adapted from FAIRbioinfo 2021 training material of the IFB

Material under CC-BY-SA licence

On schedule today:

• introduction to workflows
• introduction to Snakemake & the concept of rules
• Snakemake & SnakeFiles
• Illustration with a 2-step workflow example

What is a workflow?

Workflow = a set of instructions or operations used to complete a process (e.g. fill, close & label a bottle):

• input: empty bottle
• step 1: fill bottle
• step 2: put lid on bottle
• step 3: stick on label
• final output: filled, labelled & closed bottle

What is a workflow?

Workflow = a set of instructions or operations used to complete a process (e.g. fill, close & label a bottle):

In bioinformatics, bottles are data (i.e. an analysis of “input data” to get the final results thereof: “output data”):

Why use a workflow management system?

The pros:

• minimise the number of manual steps in an analysis
• simplify pipeline development, maintenance, and use, by dealing with:
  • task parallelisation & efficient use of resources
  • resuming of failed runs or steps
  • tracking of parameters and tool versions (=reproducibility)
• make your code:
  • less complex to read and more modular
  • more easily scalable to large sets of data
  • more transportable onto different systems (local PC, HPC or cloud)

The cons:

• learning effort…

What workflow management systems?

Many workflow management systems exist & in many forms:

• command line (shell): need to script pallelisation process manually, not easy
• command line (rules): e.g. , , , …
• graphic interface: e.g. , Taverna, Keppler, …
  
  

Focus on Snakemake

Today, we’re going to learn how to use Snakemake. Its features:

• works on files (rather than streams, reading/writing from databases, or passing variables into memory)
• is based on Python (but knowing Python is not required)
• has features for defining the environment for each task (running a large number of small third-party tools is common in bioinformatics)
• easily scaled-up from desktop to server, cluster, grid or cloud environments without modifications of your initial script (i.e. develop on a laptop using a small subset of data, then run the real analysis on a cluster)

The principle behind Snakemake

Snakemake = Python (aka “snake”, a programming language) + Make (a rule-based automation tool)

Workflows are like legos:

Workflows are made up of blocks, each block performs a specific (set of) instruction(s)

1 “block” = 1 rule:

- 1 rule = 1 instruction (ideally) - inputs and outputs are one or multiple files - at least 1 input and/or 1 output per rule

Linking data flows

Rule order is not important…

execution order ≠ code order => Snakemake does a pick & mix of the rules it needs at execution

…but matching file names is key!

Rules are linked together by Snakemake using matching filenames in their input and output directives.

 A workflow example

Below is a workflow example using 2 tools sequentially to align 2 protein sequences:

In this example, we have:

• 2 linked rules: fusionFasta and Mafft
• input protein sequence files named *.fasta
• an intermediate file generated by fusionFasta named *fused.fasta
• the final output named *aligned.fasta generated by Mafft

How Snakemake creates your workflow

How Snakemake creates your workflow

How Snakemake creates your workflow

How Snakemake creates your workflow

How Snakemake creates your workflow

How Snakemake creates your workflow

How Snakemake creates your workflow

How Snakemake creates your workflow (summary)

Snakemake (Smk) steps	running path
Smk creates the DAG from the snakefile
Smk sees that the final output *aligned.fasta doesn’t exist but knows it can create it with the Mafft rule
Mafft needs files matching *fused.fasta (don’t exist) but the fusionFasta rule can generate it
fusionFasta needs .fasta files

How Snakemake creates your workflow (summary)

Snakemake steps	running path
.fasta files exist! Smk stops backtracking
Smk runs the fusionFasta rule
P10415_P01308_fused.fasta exists and feeds the Mafft rule
the final output (P10415_P01308_aligned.fasta) is generated, the workflow has finished

Rules are run when outputs are missing… but not only

Snakemake’s job is to make sure that everything is up-to-date, otherwise it (re-)runs the rules that need to be run…

Rules are run if:

• output doesn’t exist
• output exists but is older than the input
• changes detected in parameters, code or tool versions since last execution

The Snakemake world

Many default files constitute the “Snakemake system” & there are standards on how to organise them.

They are not all necessary for a basic pipeline execution.

The most important is the Snakefile, that’s where all the code is saved.

For more information: https://github.com/snakemake-workflows/snakemake-workflow-template

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

=> Rules usually have a unique name which defines them

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

=> Rules usually have a unique name which defines them
=> input, output, shell etc. are called directives

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

=> Rules usually have a unique name which defines them
=> input, output, shell etc. are called directives
=> "myInputFile" & "myOutputFile" specify 1 or more input & output files

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

=> Rules usually have a unique name which defines them
=> input, output, shell etc. are called directives
=> "myInputFile" & "myOutputFile" specify 1 or more input & output files
=> shell specifies what to do (shell commands in this case -> alternative directives exist)

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

=> Rules usually have a unique name which defines them
=> input, output, shell etc. are called directives
=> "myInputFile" & "myOutputFile" specify 1 or more input & output files
=> shell specifies what to do (shell commands in this case -> alternative directives exist)
=> {input} & {output} are placeholders & are replaced by input & output file names at execution

Within the Snakefile…

• The Snakefile is where rules are defined
• The basic syntax of a rule is:

rule myRuleName:
____input: "myInputFile"
____output: "myOutputFile"
____shell: "echo {input} > {output}"

=> Rules usually have a unique name which defines them
=> input, output, shell etc. are called directives
=> "myInputFile" & "myOutputFile" specify 1 or more input & output files
=> shell specifies what to do (shell commands in this case -> alternative directives exist)
=> {input} & {output} are placeholders & are replaced by input & output file names at execution

=> code alignment (=indentations) is important
=> files and shell directives should be given within quotes (', " or """ for multi-line code)
=> additional & optional directives exist, e.g.: params:, resources:, log:, etc. (we’ll see them later)

For more information: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Snakefile of the previous example

rule mafft:
    input:
        "P10415_P01308_fused.fasta",
    output:
        "P10415_P01308_aligned.fasta",
    shell:
        """
        mafft {input} > {output}
        """

rule fusionFasta:
    input:
        p1="P10415.fasta",
        p2="P01308.fasta",
    output:
        "P10415_P01308_fused.fasta",
    shell:
        """
        cat {input.p1} {input.p2} > {output}
        """

How to run a Snakemake pipeline?

When Snakemake is installed (how to install):

1. move into the directory containing the Snakefile
2. type snakemake --cores 1 to run the pipeline (--cores specifies the number of cores to use)

Snakemake’s monolog & it’s hidden treasure chest

When you run Snakemake, you’ll get a full report printed on the screen of its progress:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job        count
-------  -------
fusionFasta    1
mafft          1
total          2

[...]

5 of 3 steps (100%) done
Complete log: .snakemake/log/2024-02-20T150605.574089.snakemake.log

When it’s finished, a .snakemake folder will appear in your working directory:

• it can be heavy (when using environments)
• it can contain a lot of files (unsuited for some file systems)
• it’s a hidden folder so ls -a to see it
• don’t forget to remove it once you’re sure you’ve finished your analysis

Useful debugging options

Visualise the Snakemake DAG

To visualise the complete workflow (--dag), rule dependencies (--rulegraph) or rule dependencies with their I/O files, in dot language. Uses the dot tool of the graphviz package to create a png, pdf or other format:

snakemake --dag | dot -Tpng > dag.png
snakemake --rulegraph | dot -Tpng > rule.png
snakemake --filegraph | dot -Tpng > file.png

Use the --dry-run option

Using this option will perform a “dry-run” i.e. nothing will be executed but everything that would’ve been run is displayed on the screen

Other useful options for debugging when running Snakemake

• print the shell command that is run: -p --printshellcmds
  
• print a summary and status of rule: -D

All command line options: https://snakemake.readthedocs.io/en/stable/executing/cli.html#all-options

Conclusion

So far, we’ve seen

• Snakemake workflow = set of rules
• Rules are written in Snakefiles
• Snakemake links rules together by matching up common input/output files
• Rules are defined by their name and contain directives (of which input and output to specify input & output files):

rule myRuleName
    input: "myInputFile"
    output: "myOutputFile"
    shell: "echo {input} > {output}"

• Access input & output values with {input} & {output} placeholders within the shell directive
• A Snakefile is run with the snakemake --cores 1 command (+ other options available)
• Debugging options: --dag, --rulegraph, --filegraph and --dry-run

Conclusion

Useful links

• Snakemake’s FAQ: https://snakemake.readthedocs.io/en/latest/project_info/faq.html
• Snakemake’s docs: https://snakemake.readthedocs.io

Where to get pipelines?

• complete workflows from Snakemake workflow catalog or colleagues (…or Nextflow catalog!)
• workflow bits with wrappers or inclusion
• code from scratch

A quick side note

rule fusionFasta:
    input:
        p1="P10415.fasta",
        p2="P01308.fasta",
    output:
        "P10415_P01308_fused.fasta",
    shell:
        """
        cat {input.p1} {input.p2} > {output}
        """

rule fusionFasta:
    input:
        p1="{upid1}.fasta",
        p2="{upid2}.fasta",
    output:
        "{upid1}_{upid2}_fused.fasta",
    shell:
        """
        cat {input.p1} {input.p2} > {output}
        """