Putting an NGS pipeline together

The shape of it

A pipeline isa series of file conversions

Each step reads one format and writes another. Once you see it that way, an NGS workflow stops being intimidating and starts being bookkeeping.

In: FASTQ

Reads and quality scores, gzipped, usually a pair of files per sample.

Through: BAM

Aligned, sorted, indexed, deduplicated. The working format for everything in the middle.

Out: VCF, or a count table

Small enough to take home. Almost everything else on the machine is intermediate.

The steps

Six stages,the same every time

DNA or RNA, human or bacterial — the frame is identical and only the tools inside it change.

1 · QC the raw readsFastQC, then MultiQC across samples. Catching a bad library here saves the whole downstream run.

2 · Trim, if neededAdapters and low-quality tails. Aligners soft-clip well, so trim because QC said so, not by habit.

3 · AlignBWA-MEM or Bowtie2 for DNA, STAR or HISAT2 for RNA. Pipe the output straight into a sort.

4 · Sort, index, mark duplicatesSAMtools throughout. PCR duplicates are flagged rather than removed, so the caller can decide.

5 · Call, or countVariants to a VCF for DNA; reads per gene to a count table for RNA. The fork between the two experiments.

6 · Look at itIGV over the interesting loci. A pipeline that has never been eyeballed is a pipeline you cannot defend.

Running it

A shell script worksuntil it does not

Every pipeline starts as a list of commands. The question is when to stop and pick up a workflow manager.

the minimum viable pipeline

$ fastqc -t 8 -o qc/ raw/*.fastq.gz

$ bwa mem -t 8 ref.fa r1.fq.gz r2.fq.gz | samtools sort -@ 4 -o aln.bam

$ samtools index aln.bam && samtools flagstat aln.bam

A shell script

Right for a handful of samples. Version it, and it is perfectly reproducible.

Nextflow or Snakemake

Resume from failure, skip finished steps, run samples in parallel. Worth it past ten samples.

nf-core

Community pipelines already assembled and validated. Start here before writing your own.

The honest threshold is roughly: more than ten samples, or any step that takes long enough that you mind rerunning it. Below that a script is fine and a workflow manager is overhead. Above it, resuming from a failure and not recomputing finished steps is worth the learning curve on its own.

What it costs

Disk, then memory,then cores

NGS work is bounded by different things at different steps, which is worth knowing before you pick a machine.

CoresAlignmentscales nearly linearly with threads

MemoryAssemblyhundreds of GB, and few cores

DiskAll of itbilled until you delete it

Alignment wants cores and a moderate amount of memory. Assembly wants an enormous amount of memory and does not care much about cores. Everything wants disk, and intermediate SAM files are the usual reason a volume fills up — pipe the aligner straight into samtools sort rather than writing SAM to disk at all.

Because disk here is billed from the first run until you delete it, deleting intermediates as you go is not tidiness — it is the largest single lever on what the analysis costs.

The status

BWA isbeing built

The image and its definition are being verified. When they are done, it deploys by name like the rest.

Ask for itSay so on the community tracker, and it moves up the list.Or build itOpen a pull request on the hub, following a published one.

Read it, then run itPrepaid and by the hour, on a machine that is yours about a minute after you ask.

Start a machine All notes