Proposal: Runtime images — bring your own image for a computing unit

[aicam]

Summary

A runtime image is the image a computing unit runs, built from a Dockerfile the user owns.
A computing unit started from one runs that image instead of the deployment's default, so a
workflow can use interpreters, system packages and compiled libraries the default image does
not have.

We have a working prototype and would like opinions before it goes any further — especially
on the questions at the end.

This is the same problem area as #7475, approached from the computing unit rather than the
operator.

The problem

Today a user's only way to add a dependency is a Python virtual environment: a list of pip
specs installed into a venv on the computing-unit image's own interpreter. Three common needs
fall outside that definition, and no amount of effort within the mechanism reaches them:

Need	Why a venv cannot	An image can
A different Python version	venv is built from the image's interpreter	FROM a different base
A system package (hmmer, ffmpeg, a driver)	a venv holds Python packages only	apt-get install as root, at build time
A package that compiles from source	there is no build step, only pip install <spec>	any RUN you like

A shell on a running pod does not close the gap either. That shell is the unprivileged
texera user, so apt install fails, and anything it did install would live in the
container's writable layer and vanish on the next restart.

The case that forced it: AlphaFold 3

AlphaFold 3 is a sharp example because it
fails a venv in all three ways at once — it needs Python ≥ 3.12 (the engine's is 3.10),
it is not on PyPI and compiles a C++ extension, and it calls jackhmmer/nhmmer,
which are executables, not Python packages.

This is not a hypothetical dependency. It is the shape of most scientific tooling our users
bring, and each row above is a hard stop rather than an inconvenience.

The prototype

Runtime Images in the sidebar → New runtime image → the editor opens pre-filled with
the computing-unit image's own Dockerfile, so the starting point is what already exists rather
than a blank file. Saving starts a build; Logs is readable during it and after. When
creating a computing unit, the runtime image is picked from a dropdown — only READY ones
appear.

flowchart LR
  D["user Dockerfile"] --> J["BuildKit Job<br/>(rootless)"]
  J -- "push &lt;registry&gt;/&lt;id&gt;:&lt;n&gt;" --> R["in-cluster registry"]
  R -- "kubelet pulls" --> P["computing unit pod"]

Loading

Four decisions worth commenting on:

• BuildKit, rootless. Kaniko was archived in June 2025. Rootless because a build runs
  arbitrary user-supplied instructions, and a privileged builder would make the cluster's
  isolation depend on the Dockerfile being well behaved.
• The tag carries a build number. A rebuild publishes :n+1 rather than overwriting
  :n, so a computing unit already running an earlier build is untouched.
• Status is reconciled on read. A build finishes on the cluster, not in the service, so a
  row learns its outcome when someone lists or opens it — no background threads, no leader
  election.
• Nothing is migrated. Python virtual environments still exist and still work for per-UDF
  selection.

It runs

AlphaFold 3's data pipeline, on minikube, 2 CPU / 4 GiB, no GPU — AF3 3.0.5 built from
source, calling the real jackhmmer through alphafold3.data.tools.jackhmmer, on Python
3.12.14, from an engine whose own interpreter is 3.10:

protein	residues	msa_depth	search_seconds
ubiquitin	76	466	0.075
NEDD8	81	459	0.274
SUMO1	101	322	0.297

Flipping the UDF back to the default Python environment fails the identical workflow with
No module named 'alphafold3', which is the point — the selection is load-bearing, and what
it selects is something no venv could have built.

(Structure prediction is out of scope here: no GPU, no model parameters, no 630 GB databases.
This is the dependency problem, not an AF3 port.)

Known limits of the prototype

• The registry is plain HTTP with no authentication; it works only because container
  runtimes treat the Service CIDR as insecure.
• Nothing constrains what a Dockerfile may do. An image that omits USER texera runs as
  root in the pod, because the pod spec pins no securityContext of its own.
• No quota on how many runtime images a user creates, how large one may be, or how long
  old tags are kept.
• Deleting a runtime image leaves its pushed layers in the registry.

Questions

1. Is the computing unit the right granularity? Running an operator inside a container #7475 asks for a container per operator.
   Per-unit is much cheaper to build and reason about, but one workflow then cannot mix two
   incompatible dependency sets. Is that limit acceptable, or is per-operator the real target?
2. Should users be able to name an existing image (one on Docker Hub, or any bare image
   reference) instead of authoring a Dockerfile we build? That skips the builder entirely, at
   the cost of any control over what is inside.
3. Who is allowed to create one? A build is arbitrary code execution with network access
   on the cluster. Admins only, an allowlist, or every user with quotas?
4. Do Python virtual environments stay? They are strictly weaker, but simpler for the
   common "just add a pip package" case, and they select per-UDF rather than per-unit.
5. Sharing. Runtime images are currently private to their owner. Should they be shareable
   like datasets and workflows, or published to a curated library?

[chenlica]

This proposal was based on several offline discussions between @aicam @tanishqgandhi1908 and myself, motivated a real use case of Alpha Fold 3.

This proposal includes 1) how to build and maintain an image; and 2) how to use an image for a CU. The first task is what tools such as DockerHub do, so we should avoid inventing the wheels. So I believe we should focus on 2). We can just let a user specify a URL to a DockerHub image when creating a CU, without considering how it's created.

The following are my corresponding thoughts on those questions:

1. Yes

2. Not a focus of this work

3. Not a focus of this work

4. PVEs co-exist with this feature.

5. Not a focus of this work

[parshimers]

I think this is a really great and natural idea. One of the original things that made containers so popular is being able to more effectively manage complex environmental and library dependencies.
Are there any PRs or patches in progress for this? It would be nice to see a concrete example of how some of this might work.

[tanishqgandhi1908]

The end-to-end prototype is in progress, will share a demo video and architecture once it's ready. Can start raising PRs based on feedback.

[aicam]

We decided to limit the scope of this work for the first version. We will add a new option in admin page so admin can add new CU images to the available Docker images CU-manager can use to start a new CU.
In this context, admin is responsible to build a valid CU image and add dependencies needed and publish it in Docker hub or any public registry and use its link to add it to available CU images user can choose to start a new CU.

[parshimers]

That sounds like a great way to implement things gradually. Since the CU image is already published it should be a reasonable task, at least at first, for an advanced user to simply make a new Containerfile with the base CU image in the initial FROM statement. Many clusters have private registries or mirrors as well, so it isn't a given that this image even has to be broadly available, just from within the cluster.

[tanishqgandhi1908]

This implements the reduced scope Ali described above with the demo video at the bottom walks through it.

User experience - the admin

• Go to Admin→ CU Images, fill in two boxes, and a link to the image.
• The link can be a Docker Hub page address copied from the browser, or a plain owner/name: 1.0.
• The row shows Pending → Mirroring → Ready (or Failed), and a Log button shows the copy happening line by line.
• Expect about 5 minutes for a real image - we measured ~4.3 GB on one.
• If something's wrong - tag doesn't exist, image is private, not a Texera image - the log says so plainly and tells you what to do instead.

User experience - user using models

• Creating a computing unit now has an image dropdown. Pick one, or ignore it and get the standard one.
• The choice is made when the unit is created and stays for its life. You can't swap an image under a running unit.

Architecture

• The cluster runs its own small registry. It has a fixed address, and its storage survives restarts - otherwise every image would need re-copying after a reboot.
• Copying is a background job. A tool called skopeo copies registry-to-registry. No download-and-repack, so it stays fast and unprivileged. That job's output is the log you see.
• The image is checked before any of it is copied. We read just the image's config, a few kilobytes, so a wrong link fails in seconds instead of after gigabytes.
• Each copy gets its own tag. Refreshing an image publishes a new one and leaves the old alone, so units already running keep the exact bytes they started with.
• Images are stored in pieces, and a piece the registry already holds is neither fetched again nor stored again. So a second image built on the same base costs only its own layers - in disk and in time.

What happens when a unit is created

• The image is checked first, before anything is saved. It must be Ready, and the registry must really still have it.
• The image is simply substituted into the pod — one container, one image. Nothing else about the unit changes: same limits, same permissions.
• The container runs as a non-root user, and that's pinned by the pod, not left to the image to decide.
• The unit remembers the image's name, not just its ID - so if the image is later deleted, "what is this running?" still has an answer.
• If anything goes wrong mid-creation, the half-made pod is cleaned up so nothing is left stranded.

For database: one new table: cu_image

Please find the demo video below to understand user experience

Please let me know if anything here needs changing, or if there's anything you'd want covered before I open a PR.

Demo Video:

alpha.fold.demo.mp4

[parshimers]

I mean adding a new registry to the cluster entirely is one way to do it. I would be curious to see the WIP PR if it's fine to share. I sort of thought it would be even more basic- just basically adding a setting so that the CU manager can request some other tag than the existing one (e.g. texera-workflow-computing-unit-1.2.0 or whatever).

[tanishqgandhi1908]

Sure, let me share once it's ready!

[chenlica]

@aicam and @tanishqgandhi1908 : I would recommend we first agree on the design before doing a prototype, not the other way around.