Repository navigation
Restored children share one writable rootfs — concurrent sandboxes corrupt each other #317
Description
Restores are not safe to run concurrently against one snapshot, and the failure is silent corruption rather than an error. Filing with the evidence because it cost us ~12h to attribute, and because the fix is a design decision I would rather ask you about than assume.
Mechanism
- An rw bake boots the parent with
rootfs_read_only: false(BootConfig::ext4_rw,crates/forkd-vmm/src/lib.rs:159; the ro variantext4_roat:139setstrue). - Firecracker serialises the drive's
path_on_host+is_read_onlyinto the vmstate, and reopens that path verbatim on/snapshot/load. snapshot.jsonpins the rootfs as an absolute shared path, not per-snapshot or per-child:
{ "vmstate": ".../elixir-release/vmstate",
"memory": ".../elixir-release/memory.bin",
"rootfs": "/var/cache/forkd/elixir-release-tools-local.ext4" }restore_many_withcopies/layers memory only — the v0.5 chain assembly stagesmemory-assembled.bin; the rootfs path is untouched.
So every restored child of a tag opens the same ext4 read-write. Two children, two guest kernels, one filesystem, no coordinator:
$ lsof | grep firecrack
firecrack 2588644 root 30u REG 252,1 12884901888 4628036 /var/cache/forkd/elixir-release-tools-local.ext4
firecrack 2600514 root 30u REG 252,1 12884901888 4628036 /var/cache/forkd/elixir-release-tools-local.ext4
Same inode, both 30u (read-write).
Observable damage
Random, different victims each run, and the base ext4 still hashes byte-identical to its source image in the same session — so nothing points at the image:
/usr/bin/unameholding another file's bytes — a valid ELF of plausible size, so an "it still executes" check passes; only an identity check catches it (uname --version→uniq (GNU coreutils) 9.1)./usr/bin/trlikewise.libapt-pkg.so.6.0: invalid ELF header,libidn2.so.0: cannot dynamically load position-independent executable.EBADMSGon/var/lib/apt/lists/*and/var/lib/dpkg/triggers/File— corrupt directory entries.- The ext4's mtime changing while no bake is running.
Two consequences worth separating:
- Re-baking "fixes" it until the next concurrent job, which is what makes it look like a bad image rather than a shared writer. Restarting the service does not help either: long-lived pooled sandboxes are separate from the runner process, and it re-attaches to them, so the pool is the stale-content holder and nothing invalidates it.
- ENOSPC is an amplifier, not a second cause. On a full filesystem the diff/CoW writes fail mid-block, which is why freeing disk appeared to fix it and why it returns the moment concurrency does.
On the documented non-goal
DESIGN-v0.5-diff-snapshot-chains.md (Non-goals) says chains apply to memory.bin only and that "the rootfs gets the apt / pip-installed bytes via the install step's writes to ext4; we keep copying the rootfs whole; filesystem CoW (overlayfs / btrfs) is a separate piece of work". The stated safety assumption — the rootfs is copied whole — is not what restore_many_with does; it copies nothing. Per-child FS CoW was deferred as an optimisation, but with n-way restore it is load-bearing for correctness, which moves it in priority rather than scope.
Worth noting the same root cause explains two other reports: from-image has no content identity to key a cache on (the tag string is all there is), and the warm pool has no key to evict against — so cache staleness and pool staleness are the same missing identity.
Options
1. Read-only base + per-child writable upper (my preference). Flip the bake to ext4_ro, give each child a private upper (tmpfs, or a small per-child ext4/erofs overlaid on the ro base), and let callers pass a capacity hint so a build can size its layer. No host-side per-spawn copy, so spawn latency is unchanged and disk stays bounded by the pool rather than by accumulated writes.
2. Host-side per-child reflink. reflink_copy the base per child (12.88 GiB apparent / ~5 GiB allocated here, so near-instant on XFS/btrfs), or a dm-snapshot/qcow2 diff. More general — writes land in a real file you can inspect and GC — but it materialises per spawn and costs a lot of disk at pool scale.
3. Interim invariant. Refuse to let a second live child of an rw-rootfs snapshot exist. I want to flag the cost honestly rather than present it as free: from-image prints fork --tag X -n N --per-child-netns with rw hardcoded, and restore_many_with is built to share the inode across n children, so this rejects n > 1 on rw snapshots — a behaviour break, not just a guard. It also needs the drive's ro/rw state at restore time, which Snapshot does not persist today (the vmstate carries it, but forkd never parses vmstate).
Asks
- Which option, or a different shape? I am happy to write the DESIGN amendment and implement, and I would rather have that decided before producing a diff you then have to redirect.
- Independently of the choice: should
Snapshotpersist the drive's read-only flag? Nothing in forkd metadata records it today, so any guard or diagnostic has to infer it from the.ext4extension — and the daemon's bake path records no rootfs at all (snapshot_toreturnsrootfs: None).
Until one of these lands the only workable mode on a shared-rw tag is to keep a single writer, and nothing enforces that today. The caps that approximate it (a runner-side concurrency limit, a warm pool of one) cover only runner-driven sandboxes: anything spawning the same tag through the backend API, a CLI, or an agent is an extra writer regardless, so the rule is discipline rather than enforcement and I would not call it safe.
Activity
jrimmer commentedon Sep 11, 2026
Correction to my own Option 2 — it is not available
Following up on the options I listed above: Option 2 (host-side per-child reflink) cannot be built on the current restore contract.
Firecracker serialises the drive's
path_on_hostandis_read_onlyinto the vmstate and reopens both verbatim at/snapshot/load. Neither is overridable at load time — I searched upstream v1.12 fordrive_overridesand there is no such field, and forkd's JSON path rewriting cannot reach the vmstate because it is a binary blob rather than JSON. So the host cannot point a child at its own copy: all children necessarily open the same path with the same flags. Host-side layering needs a Firecracker feature first (a drive-override on load, or a documented way to rewrite the frozen drive entry).That leaves Option 1, and it has the same shape of problem one level down: the guest cannot tell whether it is a parent or a child. The kernel command line is frozen in the vmstate along with the drive, so a guest-side overlay cannot be conditioned on "I am a restored child" by boot arguments. It needs some other channel — a host query from the guest's init, a convention on the address it was given, or mounting the base read-only unconditionally and moving the parent's warmup writes somewhere that survives.
The third of those is the one I would want your read on, because it decides whether warm caches survive at all: if the base is always read-only with a tmpfs/private upper, the parent's warmup writes (cargo registry, pnpm store,
mix deps) land in RAM and are lost with the VM — which is most of the value of the warm pool.What I have opened in the meantime
#318 records
rootfs_read_onlyinsnapshot.jsonand warns (or refuses, underFORKD_REFUSE_SHARED_RW=1) when a restore would add a second writer to a writable-rootfs snapshot. It is the part every option needs and is safe on its own: purely additive to the format, and the refusal is opt-in precisely because it rejectsfork -n N.fix(controller): detect and warn on a shared writable rootfs
jrimmer commentedon Sep 11, 2026
Consolidating the ask — one decision now
My correction above removes Option 2, and the metadata half of my original ask #2 is landed as #318, so this reduces to a single question about the guest side.
The constraints are:
/snapshot/loadhas nodrive_overridesto override either. So the host cannot give a child its own rootfs.The second constraint is what everything hinges on: a restored child has to know it is a child before it writes anything, because by the time anything else could tell it, boot has already written to the shared ext4.
One thing also falls out of the first constraint that narrows the design a lot: a tmpfs upper is the only viable per-child writable layer. Anything file-backed needs a per-child host path, and the path is frozen in the vmstate, so it would be shared again. That means a child's upper is RAM either way — the options differ only in what the parent does.
A — the guest always overlays. The base is mounted read-only and every boot takes a private tmpfs upper, parent included. No role signal is needed and it is safe by construction. Correcting my first draft: this does not lose the warm caches. A tmpfs upper lives in guest memory, and forkd's snapshot captures guest memory, so the parent's warmup writes (cargo registry, pnpm store,
mix deps) are present in every restored child. The real cost is that writable space is bounded by guest RAM, and the pages a job dirties inflatememory.bin, so each snapshot is bigger on disk and slower to restore. In exchange it retires a second problem for good: with the base never written there is no dirty ext4 or journal replay at restore, which is #296's entire failure mode.B — the guest learns its role. The parent keeps writing the base directly as it does today; a restored child mounts it read-only over a private tmpfs upper. The only thing this buys over A is that the parent's warmup bytes land on disk instead of in guest RAM during the bake. It pays for that with a boot-time protocol on the child path, where a wrong answer means a child writes the shared ext4 — the corruption being fixed.
Which do you want, A or B? My recommendation is A, and if RAM turns out to be the binding constraint I would rather add B's role signal later as a deliberate optimisation with a fail-closed default (assume child unless the host says parent) than have it on the critical path from the start.
Either way I will write the DESIGN amendment and implement it, and I am not sending the implementation until you pick: A and B share almost no code, so a guess costs you a review.
jrimmer commentedon Sep 11, 2026
Going with A, implemented as #319.
The justification, short version: the host cannot hand a child its own rootfs (frozen path and flag, no
drive_overrides) and cannot tell the guest which role it has (frozen command line), so the only property left to enforce is that the shared ext4 is never written. A enforces exactly that and needs no role signal. B buys one thing over A — the bake's warmup bytes land on disk instead of in guest RAM for the duration of a bake — and pays for it with a boot-time protocol on the path where a wrong answer means a child writes the shared ext4.I also corrected my framing of A above: it does not lose the warm caches. The upper is guest memory and
memory.binis the snapshot, which is the property the existing/tmptmpfs comment already depends on. That comment turned out to be the design sketch for this change — it is the same reasoning applied to the rest of the filesystem.If RAM turns out to be the binding constraint in practice I will come back with B as a deliberate optimisation, fail-closed, rather than have it load-bearing from the start.
jrimmer commentedon Sep 12, 2026
Correction: the constraint I built the recommendation on is wrong
I claimed there is no way to re-point a drive at restore, and used that as the load-bearing premise — first to argue A over B, then as "the host cannot give a child its own rootfs". That claim came from grepping
drive_overridesin the tree here, which is the vendored fork pinned to v1.12, plus the v1.10.1 on our host. Both predate the feature:drive_overridesto the snapshot load API — overriding a block device's host path at restore, following the same pattern asnetwork_overridesandvsock_override, closing #4014. Its motivating example is container runtimes using devmapper, i.e. exactly this case.PATCH /drives/{drive_id}rebinding works on a restored and resumed VM as of v1.15, so a placeholder drive baked into the snapshot can be pointed at a per-fork backing after restore without any load-time override.So a restored child can be given its own disk. It is a version bump plus a bake-time policy, not a missing capability.
What follows for this issue:
memory.bin— were paid for nothing. I have withdrawn that PR.cp --reflink=alwaysfor simplicity, or dm-thin with the read-only image as an external origin for scale (unwritten blocks served from the shared image, only writes land in the pool). Both exist in production — firecracker-containerd's devmapper snapshotter and harmont-dev/hyper's block stack.is_read_onlyis baked at snapshot time andPATCH /drivescannot flip it — so the read-only decision belongs at bake.What still stands from this issue, unchanged: the shared writable ext4 is real, the two-children-one-inode evidence is real, and the corruption signatures are real. Only my explanation of why it could not be fixed was wrong.
One question worth checking before rebasing the fork, and the only thing that could still make this awkward: does a snapshot taken after a
PATCH /drivesrecord the patched path? If it does, everything composes; if it records the original, per-fork drives need flattening at branch time.Status: #319 withdrawn, #320 closed (it was filed in the wrong tracker), and the tracking issue now lives in our own repo.