DiPhyxDiPhyx
Workspace
What the field is

Biology arrivedas a data problem

Sequencing got cheap faster than anything else in science. Bioinformatics is what happened when the reading outpaced the reasoning.

The cost collapsed

A genome went from a decade and billions to a day and a few hundred. Nothing else moved like that.

The bottleneck moved

Generating data is now the easy part. Storing, analysing and interpreting it is the hard part.

It is a joining field

Biology, statistics and computing, and you cannot skip any of the three for very long.

The questions

Four thingspeople actually ask

The tooling is vast; the questions underneath it are not. Almost every project is one of these.

What is different hereVariants between individuals, tumour against normal, treated against control. Comparison is most of it.
How much of this is thereExpression, abundance, coverage. Counting, then testing whether a difference in counts means anything.
What does this sequence doAnnotation, homology search, domain prediction. Reasoning from similarity to function.
What shape is itStructure prediction and docking, which is where the field meets computational chemistry.
How the work goes

Mostly moving filesbetween programs

Bioinformatics in practice is less algorithm design than careful plumbing, and the plumbing is where results are lost.

Formats, constantly

FASTQ to BAM to VCF to BED, with a conversion at nearly every step and an index beside most of them.

Conventions that disagree

Zero- against one-based coordinates, chr1 against 1. Silent, and the commonest source of wrong answers.

Sanity checks between steps

Read counts, flagstat, a quick look in IGV. Cheap now, and the alternative is finding out much later.

Each tool has an opinion about formats, coordinate conventions and chromosome naming, and none of them agree. A pipeline that runs cleanly to completion can still be silently wrong because one step used zero-based coordinates and the next assumed one-based. Checking intermediate output is not optional caution — it is the job.

Getting started

Three skills,in this order

The learning curve is real, but it is not evenly distributed. These three carry most of the weight.

The shellFirstevery tool is a command
R or PythonThenthe statistics, and the plots
Version controlThirdan analysis you cannot rerun is an anecdote

The shell comes first, because every tool is a command and every pipeline is a chain of them. Then R or Python for the statistics, since the interesting part of most projects is a differential test rather than an alignment. Version control comes third and matters more than people expect: an analysis you cannot rerun is an anecdote.

Most of the time is not computeIt is finding the right reference, fixing a format, and working out why two tools disagree.
Look at your dataPlots and genome browsers catch things no summary statistic will. Do it before you trust a p-value.
Compute in burstsAlignment and assembly want a big machine for hours; the rest of the project wants a laptop.
On a machine

SAMtools isalready built

Nothing here needs compiling. Pick a machine, and these two commands put it on there.

hub://samtools

$ dxflow workflow create --identity samtools hub://samtools

$ dxflow workflow start samtools

Start a machineWhat SAMtools needs
Read it, then run itPrepaid and by the hour, on a machine that is yours about a minute after you ask.
Start a machineAll notes