Infrastructure

A bucket and a volume,in either direction

The AWS CLI over S3, and over anything that speaks it.Pull the inputs in before a run, push the results out after one.

AWS CLI 2.36.432 cores · 2 GBRuns as a job

hub://s3

$ dxflow workflow create --identity s3 hub://s3 --start

01Pick a machinecores, memory and a rate per hour

02Create the workflowpulled from the hub, once

03Run it, then stopstopped when the work is done

The application

Whichever sidecarries the uri

SOURCE and TARGET are the two ends, and one of them is an s3:// uri.

Down, or up

s3:// on SOURCE pulls into the volume; on TARGET it pushes back out.

S3, or anything like it

MinIO, Ceph, R2, Wasabi and B2 — set ENDPOINT and it addresses by path.

Keys at the start

Pass them on --override, so nothing is written into the definition.

How you run it

The directioncomes from the uris

Whichever side carries s3:// decides which way the data moves. Nothing else declares it.

SOURCE and TARGETOne of them is an s3:// uri and the other a path in the volume. Two local paths are refused; two buckets transfer across.

ENDPOINT and REGIONSet an endpoint for MinIO, Ceph, R2, Wasabi or B2, and addressing switches to path style on its own.

ACCESS_KEY and SECRET_KEYBoth empty means an unsigned read of a public bucket. One without the other is an error, not a fallback.

MODEsync moves what differs by size or time. copy always transfers, and takes a single object unless SOURCE ends in a slash.

DELETEOff, so sync never removes anything. Turn it on when you want a true mirror and mean it.

EXTRASplit on whitespace and never passed through a shell. Write a pattern as one word, unquoted.

pull a dataset in

$ dxflow workflow start s3 --override env.job.SOURCE=s3://my-bucket/runs/ --override env.job.TARGET=/volume/input

$ dxflow workflow start s3 --override env.job.ACCESS_KEY=AKIA... --override env.job.SECRET_KEY=...

The job

It transfers,and then it stops

A job, not a session — it ends once the last object has landed.

s3://Sourcea bucket prefix, or a path

/dataTargetthe volume it lands in

sync, or copysync moves what differs, and MODE=copy takes a single object.

Watch it from the CLIdxflow workflow logs names each object as it transfers.

Good to know

It mounts the wholevolume, on purpose

That reach is what lets a fetch land in another workflow's directory. It is also the thing to be careful with.

It lands where the next run looks

The volume is the root of Artifacts, so /volume/input inside the container is the same directory another workflow already reads.

Do not upload the root

SOURCE=/volume on a push sends every workflow's folder, workflow.json and the logs to the bucket. Point it at a directory inside.

Keys belong on the start line

Pass them with --override, scoped to the run. The hub entry is public, and a definition is not the place for a secret.

The image

Pulled once,then it stays

S3 Sync arrives as one image. This is what comes down the first time, and what the disk should have free for it.

134Mamd64compressed, the way the registry counts it

130Marm64compressed, the way the registry counts it

10GOn diskunpacked, with room to work beside it

ghcr.io/dxflow-ai/s3:latestPublished from the hub, pulled on the first start and kept for the ones after it.

Either architectureamd64 and arm64 are both published, and the machine pulls the one it runs.

What it asks for

What it wants,and what it needs

The definition asks for 2 cores and 2 GB. The image comes up on less than that, and a start given --fit trims the ask to whatever the machine actually has.

2 cores · 2 GBAsks forwhat the definition writes down

1 cores · 1 GBRuns onthe least the image comes up on

Not neededGPUit works on the cores alone

The ask is not the floorThe definition writes down what suits the work. The image itself starts on less, which is what the second figure is.

--fit caps it to the hostA start given --fit trims each step to what the machine actually has, for that start alone. The definition is never rewritten.

Machines that fit it

S3 Sync asks for 2 cores and 2 GB. Cheapest first.

E2 Small

$0.0209/ hour2 cores · 2 GBStart this machine

E2 Medium

$0.0419/ hour2 cores · 4 GBStart this machine

T3 Medium

$0.052/ hour2 cores · 4 GBStart this machine

Run S3 Sync on your own machinePick a machine that meets it, and it opens about a minute after you ask.

Start a machine Browse workflows