Infrastructure

The volume,in a browser tab

A multi-user file manager over the engine volume.Upload, unpack and stage a run without a client.

FileGator 7.16.22 cores · 2 GBServed on 8080

hub://filegator

$ dxflow workflow create --identity files hub://filegator --start --link

01Pick a machinecores, memory and a rate per hour

02Create the workflowpulled from the hub, once

03Run it, then stopstopped when the work is done

The application

The data moveswhere it already is

The directory it browses is the one Artifacts shows, and every workflow mounts.

Uploads that resume

It arrives in chunks, so a dropped connection continues instead of restarting.

Unzip on the engine

Unpack a reference dataset in place, without it crossing the network twice.

Stage the next run

Move one run's output into the directory the next workflow reads from.

How you run it

An account,and how far it reaches

The administrator and the guest are reconciled from the environment on every start. Everyone else you add in the UI.

USERNAME and PASSWORDThe administrator. It ships as admin and dxflow, and a start with that default publishes a writable file manager over the whole volume.

ROOT_DIRWhat the browser can see. Empty means everything, workflow.json and the other workflows included.

GUEST_PERMISSIONSAnonymous access, granted to anyone who reaches the port. read and download make a share; upload alone makes a drop box.

UPLOAD_MAX_SIZE and UPLOAD_CHUNK_SIZEThe ceiling, and the piece size it climbs in. Raise the chunk on a fast link and lower it on a flaky one.

RUN_ASroot, so it can manage the root-owned files every other step leaves behind. www-data if an unprivileged server matters more.

OVERWRITE_ON_UPLOADOff, so a second upload of the same name lands beside the first rather than on top of it.

start it your way

$ dxflow workflow start files --override env.app.PASSWORD=something-long

$ dxflow workflow start files --override env.app.ROOT_DIR=runs --override env.app.GUEST_PERMISSIONS=read|download --link

The session

One port,and who you hand it to

An account per collaborator, each with a home directory and permissions of its own.

8080Webthe file listing

/volumeStoragewhat every run wrote

A link, and no accountGUEST_PERMISSIONS opens a read-only share, or a write-only drop box.

Set a password firstPASSWORD defaults to dxflow, and everyone can read this page.

Good to know

It works wherethe data already is

Unzipping a forty gigabyte archive should not mean pulling it to a laptop and pushing it back.

Zip and unzip on the engine

An archive is unpacked under the volume, so it never crosses the network twice. Size the storage for the contents, not the file.

Stage one run for the next

Move a finished output into another workflow's input directory, and edit the parameter file in place on the way past.

The same reach cuts both ways

A browser over the whole volume can delete as easily as it can stage. Set ROOT_DIR, and give collaborators an account with a home of its own.

The image

Pulled once,then it stays

FileGator arrives as one image. This is what comes down the first time, and what the disk should have free for it.

171Mamd64compressed, the way the registry counts it

166Marm64compressed, the way the registry counts it

50GOn diskunpacked, with room to work beside it

ghcr.io/dxflow-ai/filegator:latestPublished from the hub, pulled on the first start and kept for the ones after it.

Either architectureamd64 and arm64 are both published, and the machine pulls the one it runs.

What it asks for

What it wants,and what it needs

The definition asks for 2 cores and 2 GB. The image comes up on less than that, and a start given --fit trims the ask to whatever the machine actually has.

2 cores · 2 GBAsks forwhat the definition writes down

1 cores · 2 GBRuns onthe least the image comes up on

Not neededGPUit works on the cores alone

The ask is not the floorThe definition writes down what suits the work. The image itself starts on less, which is what the second figure is.

--fit caps it to the hostA start given --fit trims each step to what the machine actually has, for that start alone. The definition is never rewritten.

Machines that fit it

FileGator asks for 2 cores and 2 GB. Cheapest first.

E2 Small

$0.0209/ hour2 cores · 2 GBStart this machine

E2 Medium

$0.0419/ hour2 cores · 4 GBStart this machine

T3 Medium

$0.052/ hour2 cores · 4 GBStart this machine

Run FileGator on your own machinePick a machine that meets it, and it opens about a minute after you ask.

Start FileGator Browse workflows