'%20d='M364.9%20143.2v98.9l-1.4%201.2-165.6%20140.4%2035.7%2029.7v-34.2l1.4-1.2%20123.4-105.4%206.5-5.6v37l-1.3%201.2-99.7%2085.8v84.7l-6.6-5.7-97-83.1-3.6-3%203.5-3%20174.4-150.6v-25.1L153.7%20360.5l-6.6%205.7v-98.9l1.4-1.2%20161.6-138-35.7-29.7v34.4L273%20134%20153.7%20236.7l-6.6%205.7v-37l1.3-1.2%2097-83.2V36.3L252%2042l97.1%2083.1%203.6%203-3.6%203-171.7%20147.7V304l106.2-90.6%2056.4-48.8%2018.3-15.7z'%20/%3e%3c/svg%3e)
Infrastructure
A bucket and a volume,in either direction
The AWS CLI over S3, and over anything that speaks it.Pull the inputs in before a run, push the results out after one.
AWS CLI 2.36.412 cores · 2 GBRuns as a job
hub://s3
$ dxflow workflow create --identity s3 hub://s3
$ dxflow workflow start s3 --override env.job.SOURCE=s3://my-bucket/runs/
01Pick a machinecores, memory and a rate per hour
02Create the workflowpulled from the hub, once
03Run it, then stopstopped when the work is done
The application
Whichever sidecarries the uri
SOURCE and TARGET are the two ends, and one of them is an s3:// uri.
Down, or up
s3:// on SOURCE pulls into the volume; on TARGET it pushes back out.
S3, or anything like it
MinIO, Ceph, R2, Wasabi and B2 — set ENDPOINT and it addresses by path.
Keys at the start
Pass them on --override, so nothing is written into the definition.
The job
It transfers,and then it stops
A job, not a session — it ends once the last object has landed.
s3://Sourcea bucket prefix, or a path
/dataTargetthe volume it lands in
sync, or copysync moves what differs, and MODE=copy takes a single object.
Watch it from the CLIdxflow workflow logs names each object as it transfers.
Run S3 Sync on your own machinePick a machine that meets it, and it opens about a minute after you ask.