Distributed systems, and when to avoid them

The definition worth using

A system wherepart of it can fail

Not many machines working together, but many machines where one can fail while the rest carry on. Everything hard follows from that.

Partial failure

One machine dies and the rest keep running. A single program either works or does not; this is neither.

You cannot tell why

A slow reply and a dead machine look identical from outside. There is no way to distinguish them.

No shared clock

Two machines disagree about the time and about the order events happened in. Both are correct locally.

What breaks

Four assumptionsthat stop being true

Things a single program takes for granted, and which cease to hold the moment a network is involved.

The network is reliableIt is not. Messages are lost, duplicated, delayed and reordered, and your code has to survive all four.

Latency is zeroA call across a network is thousands of times slower than a local one, and sometimes never returns.

Operations happen onceA retry after a timeout may execute twice. Anything that mutates state has to be safe to repeat.

Everyone sees the same thingWithout coordination, two nodes hold different views of the truth, and both are internally consistent.

The constraint

You cannot haveall three

When the network partitions — and it will — you choose between answering and being correct. There is no third option.

One truthConsistencyevery read sees the latest write

Always answersAvailabilityeven when part of it is cut off

Not optionalPartitionsthe network will split, so choose

A partitioned system can keep serving requests from both halves and risk them disagreeing, or refuse requests until the partition heals. Databases make this choice explicitly and advertise it. The mistake is assuming a system made a different choice than it did, and discovering otherwise during an incident.

Where scientific work meets it

Mostly you arenot building one

The honest position for most computational research: you consume distributed systems rather than write them, and that is a much easier job.

Embarrassingly parallel

A hundred independent runs. No coordination needed, so this is a scheduling problem, not a distributed one.

Tightly coupled MPI

Ranks exchanging boundaries every step. Distributed, but on a reliable network with a single failure domain.

Services you call

Object storage, schedulers, APIs. Somebody else solved the hard part; your job is to retry properly.

A parameter sweep across a hundred inputs is embarrassingly parallel — no coordination, no shared state, and each run is independent. That is not a distributed systems problem, it is a scheduling one, and a workflow manager or a queue solves it. Genuine distribution starts when the parts must agree on something, and if you can avoid needing that, avoid it.

Make retries safeDesign every operation so running it twice is harmless. This one habit prevents most distributed bugs.

Checkpoint long runsA week-long job with no checkpoints is a bet that nothing fails for a week. It is not a good bet.

Prefer one big machineIf the work fits on a single node, put it there. You remove an entire category of failure for free.

The status

Slurm isbeing built

The image and its definition are being verified. When they are done, it deploys by name like the rest.

Ask for itSay so on the community tracker, and it moves up the list.Or build itOpen a pull request on the hub, following a published one.

Read it, then run itPrepaid and by the hour, on a machine that is yours about a minute after you ask.

Start a machine All notes