DiPhyxDiPhyx
Workspace
The shape of a pipeline

Each format isone step's output

The formats are not arbitrary. They form a chain, and knowing where you are in it tells you which tool comes next.

Off the sequencer

FASTQ — reads and per-base quality. The rawest thing you will normally handle.

After alignment

SAM, BAM or CRAM — the same reads, now placed against a reference genome.

After calling

VCF, BED, GFF — statements about positions, small enough to open in a text editor.

Sequence

FASTA and FASTQ,and the difference is quality

Two text formats that look almost identical. One carries confidence per base and the other does not.

FASTAA header line starting with >, then the sequence. Used for references, assemblies and protein sets.
FASTQFour lines per read: identifier, sequence, a plus, and one quality character per base.
Both are usually gzippedAlmost every tool reads .gz directly. Decompressing first wastes disk you are paying for.

FASTQ quality is a Phred score encoded as one printable character per base, so the quality line is always exactly as long as the sequence line. If those two lengths disagree, the file is truncated — a surprisingly common outcome of an interrupted download, and one that tools report in confusing ways.

Alignment

SAM, BAM, CRAMhold where reads landed

Three encodings of one record type: this read, this position, this many mismatches, this confidence.

SAM

Plain text, one line per alignment. Readable, and far too large to keep.

BAM

The binary form, compressed and indexable. The working format for essentially everything.

CRAM

Smaller again, stored as differences from a reference you must keep alongside it forever.

CRAM is worth knowing about specifically because of its catch. It stores differences against a reference rather than whole reads, which is why it is so much smaller — and why the file becomes unreadable if you lose the exact reference it was written against. Archive the reference with the data, or use BAM.

Findings

VCF, BED and GFFdescribe positions

Once the reads are placed, the remaining formats all say the same kind of thing: something is here, on this chromosome, between these coordinates.

VCFVariants: a position, the reference allele, the alternatives, and per-sample genotypes.
BEDIntervals. Three columns is legal; the rest are optional. The lingua franca of region arithmetic.
GFF3 and GTFAnnotation: genes, transcripts, exons, with attributes. GTF is the older, stricter relative.
BigWig and BedGraphContinuous signal along the genome — coverage, methylation, ChIP. BigWig is the indexed one.
0-basedBEDhalf-open intervals
1-basedVCF, GFF, SAMinclusive intervals
Off by oneMixing themsilently, in either direction

The coordinate trap is worth committing to memory: BED is zero-based and half-open, while VCF, GFF and SAM are one-based and inclusive. A feature at BED chr1 100 200 is the same span as GFF chr1 101 200. Off-by-one errors between tools are almost always this, and nothing warns you.

On a machine

SAMtools isalready built

Nothing here needs compiling. Pick a machine, and these two commands put it on there.

hub://samtools

$ dxflow workflow create --identity samtools hub://samtools

$ dxflow workflow start samtools

Start a machineWhat SAMtools needs
Read it, then run itPrepaid and by the hour, on a machine that is yours about a minute after you ask.
Start a machineAll notes