fix(s3): harden burst load and large-object clones

[forhappy]

Summary

• replace the process-wide fail-fast semaphore with bounded FIFO control, read, and transfer admission pools so uploads cannot starve metadata or range reads

• retain read capacity for the lifetime of streamed GET responses, including disconnect cleanup

• add cancellation-aware backoff for high-fanout multipart part registration and return empty listings for unknown or unborn refs

• heap-pin canonical readability maintenance to prevent worker-stack exhaustion after write bursts

• commit an exact same-directory Git LFS tracking rule atomically with every large S3 object pointer, including retry-safe rule convergence

• use the upstream Git attribute parser and glob matcher so quoted, escaped, unset, and nested rules follow Git semantics

• make eager Crab clones pull historical LFS pointers even when older commits lack matching attribute rules

• plan final LFS publication against the physical backend: S3 and GCS stay within 10,000 5 GiB parts, Azure stays within 50,000 4,000 MiB blocks, and GCS rejects completed objects above 5 TiB before reading selected parts

• stream adaptive provider parts from 8 MiB payload blocks and serialize parts above the 32 MiB retained-payload budget, avoiding concurrent multi-gigabyte payload retention

• bind successful LFS uploads to the multipart completion validator so parallel S3 range readers fetch only their requested slices instead of rehashing the whole object per range

• eliminate assembled-object scratch above 64 MiB by hashing frozen multipart parts once and replaying them through a backpressured, size-and-SHA-verified LFS byte stream

• give every UploadPart transfer a unique immutable payload identity and reclaim superseded or rejected payloads without risking a concurrent winner

• reserve every backend part transfer before upload, account retired replacement payloads in the same registration CAS, and retain terminal capacity until late writers are fenced and cleaned

• surface an explicit fsck plus metadb owner recovery path when repositories written by older builds lack verified Git visibility evidence

• let a fully validated self-contained visibility proof survive retirement of its preferred catalog checkpoint without masking corrupt catalog metadata

• cancel only debounced readability work; once canonical publication starts, drain its manifest/catalog/visibility boundary and coalesce later writes into a follow-up pass

• persist per-upload expiry and staging-byte budgets, coordinate a fixed CAS-fenced capacity-slot catalog across replicas, and reconcile expired, terminal, and pre-publication-crash sessions from a bounded background sweep

• bind multipart completion to a deterministic ref-journal publication plan so immutable commit evidence recovers the original outcome even after a later write replaces the object

• add a separate internal management listener with process liveness and fresh repository-view readiness, plus CLI probes and bounded failure responses

• export private Prometheus metrics with fixed-cardinality HTTP, full response-stream, and admission-pressure series, without request or credential identities

• classify aggregate multipart maintenance as successful or degraded, expose bounded action/failure reasons and staleness, and retain per-slot failures that were previously visible only in logs

• account content-spool and Xet-reconstruction temporary files through their full ownership lifetime, with bounded pressure, throughput, and I/O-failure metrics

• qualify a digest-pinned non-root gateway image under a read-only root filesystem with RustFS, SigV4 traffic, presigned/tampered requests, range reads, and graceful SIGTERM

• add an isolated Docker Compose profile with persistent RustFS data, protected file-mounted gateway credentials, dropped capabilities, bounded scratch, explicit initialization, safe shutdown, and an opt-in volume teardown

• qualify durable multipart recovery across forced Compose gateway replacement and verify the completed object byte-for-byte

• add a fail-closed Helm chart for Kubernetes 1.29/EKS with digest-pinned images, multi-zone replicas, strict pod security, bounded scratch, private management probes, rolling-update safety, and a disruption budget

• export bounded logical object-store request, duration, in-flight, read-byte, and successful-write-byte metrics across full GET/HEAD/range/PUT/delete/list/copy and multipart lifecycles

• report scratch-filesystem total, free, and process-available bytes on every scrape, with fail-closed probe health that cannot retain stale capacity after mount loss

• atomically reserve scratch for declared and streamed request spools, Xet ranges, and generated Git packs, retaining bounded headroom and returning retryable SlowDown before disk exhaustion

• require one process-shared, explicitly placed, byte-bounded immutable-read cache; prove descriptor-relative writes before listening and mount cache separately from request scratch

• export fixed-cardinality memory/local/service cache hit, miss, failure, verified-byte, persistence-failure, and catalog usage/health telemetry without repository, object, or credential identity

• prevent protocol-library debug/trace events from exposing signed requests, signature material, or raw malformed request bodies, and enforce the boundary even under RUST_LOG=trace

• ship opt-in, release-scoped PodMonitor and PrometheusRule resources with 14 alerts covering maintenance stalls, request/backend failures, admission saturation, and scratch/cache pressure

Failure evidence

• before admission and maintenance fixes, write bursts could fail fast, 64-part completion exhausted state CAS retries, and canonical readability maintenance overflowed a Tokio worker stack

• before the listing fix, ListObjectsV2 below an unborn ref returned InternalError

• before the large-object fix, gateway-authored large objects cloned as LFS pointer text unless users ran crab lfs pull manually

• the previous hand parser misread quoted attribute patterns, causing special-character filenames to be cleaned through Crab/Xet instead of LFS

• an unrooted basename rule also affected same-named descendants; generated rules now use /filename within the nearest attributes directory

• fixed 8 MiB LFS backend parts exceeded the S3/GCS 10,000-part ceiling above about 80 GiB; a universal 5 GiB plan could also exceed the Azure 4,000 MiB block ceiling

• fresh streamed LFS uploads did not persist their completion validator; a 320 MiB AWS high-level download issued 40 ranges and redundantly read and hashed about 12.8 GiB from the backend, taking 178.17 seconds

• multipart completion previously assembled every selected part into a full-object local spool, making node scratch capacity an accidental object-size ceiling

• deterministic part payload paths made same-ETag concurrent cleanup unsafe, while replacements accumulated every older backend payload

• a process exit after backend upload but before part registration could leave unaccounted bytes, while an exit after replacement registration but before deletion could orphan the retired winner

• the reused live repository emitted a generic missing-visibility warning after every write because older qualification generations predated the current proof contract

• repaired historical proofs were rediscovered on every fsck because reads preferred catalog proofs whose exact SlateDB checkpoints had retired

• a staggered signed-write sequence exposed that cancelling active readability maintenance could advance a manifest generation before its catalog-bound visibility proof was complete

• active multipart sessions and registered staging bytes had no cross-process bound; abandoned Open uploads retained their slot and payloads indefinitely, and multipart listing scanned all historical state

• the first live expiry/restart probe found that ListParts still exposed an Aborted session after its payload cleanup; Open-only loads now make every terminal session return NoSuchUpload

• current-object attributes alone could not prove a successful multipart completion after a later overwrite; a retry could either remain fenced forever or evaluate its original condition against the later object

• per-slot multipart reconciliation errors were logged and then folded into a nominally successful sweep, leaving operators unable to alert on stalled cleanup

• content spools and Xet range reconstructions consumed local scratch without exposing live ownership, write rate, or destination-I/O failures; request-count admission alone could not diagnose volume pressure

• backend storage calls previously exposed no latency, outcome, in-flight, or byte telemetry, leaving provider saturation, credential failures, and slow range streams indistinguishable from gateway pressure

• scratch ownership metrics did not reveal unowned files, mount loss, or actual free capacity; a failed external filesystem probe could also leave stale healthy values in a dashboard

• request-count admission and telemetry did not prevent concurrent spools, reconstruction, and Git-pack preparation from racing to exhaust scratch; reservation now serializes the capacity snapshot with in-process claims

• the read-only image resolved the unlimited default cache beneath /var/lib/crab, so cache publications silently fell back to repeated provider reads; startup now rejects a missing or unwritable bounded cache mount

• bounded cache placement alone did not reveal whether traffic hit memory, local disk, or a cache service, whether local persistence failed, or whether retained catalog accounting was healthy; the gateway now exports those signals without identity labels or payload scans

• the pinned S3 protocol library logs complete signed requests and signature material at debug level and raw malformed XML bodies at error level; an operator enabling broad trace logging could expose reusable authorization material

• gateway metrics existed without a chart-owned scrape target or executable alert semantics, so saturation, scratch/cache failure, backend authorization failure, and stalled multipart maintenance could remain dashboard-only signals

Verification

• cargo test -p crab-storage --locked: 196 passed plus 2 credential-redaction integration tests

• cargo test -p crab-lfs --locked: 56 passed

• cargo test -p crab-s3-gateway --locked: 110 passed

• cargo test -p crab-cache --locked --lib --features local-cache: 228 passed

• cargo test -p crab-cache-store --locked: 46 passed

• cargo test -p crab-cache-store --locked --features remote-client: 71 passed, including exact local-miss/service-range-hit observations

• cargo test -p crab-read --locked: 163 passed

• cargo test -p crab-metadata --locked --all-features: 368 passed, 1 ignored large-repository stress test

• cargo test -p crab-remote --test publication --features publication --locked: 13 passed

• cargo test -p crab-metadata --test plan_receipt_recovery --features storage --locked: 1 passed

• cargo test -p crab-git --locked: 266 passed, 1 ignored microbenchmark

• targeted Crab eager-clone and explicit-repository LFS checkout tests passed

• cargo clippy -p crab-read -p crab-s3-gateway --all-targets --locked -- -D warnings

• cargo build --release -p crab-s3-gateway --locked

• cargo fmt --all -- --check and git diff --check

• make architecture-check: 144 direct workspace dependency edges admitted; fixes the prior Multi-crate guardrails failure

• web production build passed; link checker validated 398 HTML pages and 4,307 fragments

• live RustFS-backed gateway: 64 simultaneous 5 MiB parts completed a 335,544,320-byte object while 75 concurrent HEAD/range probes succeeded

• final-tree clone proof: 335,544,320-byte PUT, byte-identical 128-byte range, rooted escaped attribute rule, and fresh eager clone with SHA-256 9942003e84c1648820149cb7b82869eb1e6515ddd04951bd2c69f9273b09c053

• provider-aware live proof: fresh 335,544,320-byte random multipart upload, exact 1 KiB range at offset 256 MiB, no leaked scratch request directory, and full SHA-256 ac25495163208b6f448d266c595b4950b4fc08eaee2a1b5fde96e972bcf4c468

• bounded-scratch live proof: a brand-new 335,544,320-byte content hash completed from four durable 80 MiB parts in 56.64 seconds with only 6,317 bytes of peak completion scratch sampled every 5 ms; full download matched SHA-256 1ae20c0d8241a61aa2bef60b580be773bc594ccd8f28ad3924faf4b11fc083c2 and a 4 KiB late range matched exactly

• concurrent completion proof: four new 83,886,080-byte objects completed simultaneously on one branch in 17.60 seconds with 7,482 bytes aggregate peak completion scratch and 222,560 KiB peak gateway RSS; all four full SHA-256 downloads and four late ranges matched

• the rebuilt gateway retained the 320 MiB object across process restart; an 8 KiB range near 300 MiB matched in 0.515 seconds

• the same AWS 40-range full download fell from 178.17 seconds before validator persistence to 2.05 seconds after it, with an identical full-object hash

• boundary tests prove maximum-size plans remain below each provider part-size and part-count limit; live GCS/Azure transport qualification still requires CI credentials

• multipart replacement unit proof: 32 concurrent writes to one part number leave exactly the catalog winner as the only backend payload

• multipart crash proofs: in-flight bytes are durably quota-bound, a frozen session retains capacity until late cleanup, expired transfers wait through provider drain, and retired replacement payloads survive process exit for reconciler deletion

• live RustFS replacement proof: backend payload count stayed at one across replacement, completion returned the replacement bytes with matching SHA-256, and terminal cleanup reduced the part prefix to zero objects

• live signed replacement proof on the final binary: SigV4 create/upload/list/complete/GET replaced a 5 MiB part and reconstructed 6,684,672 exact bytes with SHA-256 404e6a006b7718f4c4a1616d45f6421d9ac17aef7625b59657cf2598bab1a884

• live visibility upgrade proof: a fresh repository published catalog-bound proofs at generations 1 and 2 without warnings; crab fsck --repair backfilled ten historical proofs in the reused repository, crab metadb owner --once published generation 25, and a signed write/delete advanced through generations 26 and 27 without the warning

• live historical-proof proof: before the fix, a repair reported 24 repaired and one failure but an immediate scan rediscovered 25 retired-checkpoint warnings; the fixed binary scans the same long-lived repository with errors: 0, repair_failures: 0, and passed: true

• live maintenance-race proof: after repairing the reused repository at generation 29, six staggered signed PUT/readbacks and six staggered signed DELETEs completed while canonical maintenance ran; the fixed gateway emitted no maintenance warning and the current generation retained a valid visibility proof

• repository maintenance proof: crab repack reduced the live repository from 15 active packs to 1 (6,305,434 → 6,298,270 bytes); afterward a 1 MiB range and the full 6 MiB multipart object still matched the independent source bytes

• 64 simultaneous session creates against one shared store admitted exactly 8 configured capacity slots and rejected 56

• live two-process capacity proof: two uploads admitted, the third returned SlowDown, a session created on one replica aborted through the other, and the released generation was reused

• live 10 MiB staging-budget proof: an 8 MiB part remained authoritative, a second 4 MiB part returned SlowDown, ListParts stayed at one, and only one backend payload remained

• live restart/expiry proof: the session was durably Aborted, active listing returned zero, ListParts returned NoSuchUpload, and the backend payload prefix was empty

• live historical-completion recovery proof: after a 6 MiB multipart publish was overwritten by a newer object, a fault-injected Completing record recovered from its immutable publication-plan receipt on restart, released its slot, removed all staged payloads, returned the original ETag on retry, and preserved the newer object byte-for-byte

• live signed-request proof: AWS SigV4 header auth and presigned GET/PUT succeeded, a presigned range GET returned the exact 65,536-byte slice, and tampering with either a signed query or signed Content-Type returned 403 SignatureDoesNotMatch

• live management-plane proof: the initialized long-lived repository returns 200 for liveness/readiness, an uninitialized repository remains live but returns readiness 503 with Retry-After: 5, the S3 listener rejects /livez as unsigned traffic, and SIGTERM exits zero

• live metrics proof on the rebuilt binary: signed list/PUT/HEAD/full GET/range GET/DELETE traffic produced nonzero HTTP duration and admission series; the scrape contained no bucket, key, principal, access key, or secret; management POST returned 405 and the S3 listener required a signature

• live maintenance metrics proof: a clean RustFS-backed cycle reported success, duration, a nonzero last-success timestamp, seven bounded zero-valued failure reasons and no identity; a second fault-injected backend with a corrupt capacity record reported one degraded cycle, one reconciliation error, and no false last-success timestamp

• packaged-image backend telemetry CI passed at commit b7e4778f9d4: the clean non-root/read-only image produced positive logical backend read/write counters under signed list/PUT/full GET/range GET/delete traffic, released all 11 in-flight operation gauges, retained identity-free labels, rejected tampered/wrong-secret/unsigned requests, shut down cleanly, and recovered multipart state across Compose replacement in S3 gateway run 34523590765

• packaged-image metrics CI passed at commit 5782ad33ca4: the digest-pinned non-root/read-only image passed signed list/PUT/full GET/range GET/delete; valid and tampered presigned requests; wrong-secret and unsigned rejection; positive content-spool writes; fully released scratch gauges; zero bounded scratch I/O failures; identity-leak assertions; graceful termination; and Compose replacement recovery in S3 gateway run 34519602863

• packaged-image CI passed at commit c7a1da7819a: digest-pinned build, non-root/read-only runtime, uninitialized readiness rejection, RustFS initialization, SigV4 PUT/GET/range/presign/tamper rejection, container health and graceful SIGTERM

• Compose startup, initialization, signed discovery, multipart recovery across forced gateway replacement, exact assembled-byte verification and shutdown passed in CI at 0d05f21f027

• Helm fail-closed values validation, manifest rendering, private management-port assertion, and strict Kubernetes 1.29 schema validation passed in CI at a1a07cb01a5

• the packaged Linux image accepted a Kubernetes-style root-owned, process-group-readable credential (root:10001, mode 0440), then passed initialization, signed traffic, graceful shutdown, and Compose replacement recovery in S3 gateway run 34509816529

• backend observation proof covers routed and staged stores, high-level and resumable multipart, native multi-range reads, partial-body cancellation, fixed failure classes, 99 pre-registered bounded outcome series, workflow lint, strict clippy, and a release build

• scratch-filesystem proof covers live total/free/available capacity, mount-loss zeroing, explicit probe health and failure counting, workflow lint, strict clippy, all 104 gateway tests, and a release build

• scratch-admission unit proof covers bounded headroom, exact and incremental claims, reservation release, pre-body rejection for declared sizes, bounded rejection reasons, retry headers, and conservative Git-pack peak estimates

• bounded-cache proof covers private descriptor-relative publish/sync/removal with no retained probe, explicit 64 MiB–1 PiB config validation, a fixed-cardinality configured-limit metric, Compose rendering, strict Helm lint, and strict Kubernetes 1.29 schema validation

• packaged-image bounded-cache CI passed at commit bddf02ee812: the non-root/read-only image rejected startup without its writable cache volume, created a private cache root with the configured 128 MiB ceiling, passed signed SigV4 and presigned traffic plus tamper/wrong-secret/unsigned rejection, preserved exact range reads, shut down gracefully, and recovered multipart state across Compose replacement in S3 gateway run 34531745429

• packaged-image cache-telemetry CI passed at commit 337e341c5f0: the non-root/read-only image passed signed SigV4 and presigned traffic plus tamper/wrong-secret/unsigned rejection; exported all nine fixed cache source/outcome series and three source-byte series; reported healthy catalog accounting with zero probe and persistence failures; exposed no bucket, key, principal, access key, or secret; shut down gracefully; and recovered multipart state across Compose replacement in S3 gateway run 34534587779

• packaged-image signed-request log-boundary CI passed at commit 3334726b62f: under RUST_LOG=trace, real AWS SigV4 header and presigned GET traffic, tampered presigned requests, wrong-secret and unsigned rejection, and exact ranges completed without emitting credential syntax, secrets, or reusable signatures; graceful shutdown and Compose multipart recovery also passed in S3 gateway run 34537164160

• packaged-image and Helm monitoring CI passed at commit a77a297e7c1: Prometheus Operator resources remain absent by default, opt-in rendering preserves the private management port, strict Kubernetes 1.29 validation passes, and pinned promtool syntax plus healthy/faulting semantics prove all 14 alerts; the rebuilt non-root/read-only image also passed signed/presigned/tampered traffic, exact range reads, credential-safe trace logging, graceful shutdown, and Compose multipart recovery in S3 gateway run 34538754642

• packaged-image scratch-admission CI passed at commit 9155707507f: a signed sparse 500 MiB PutObject against the gateway 512 MiB tmpfs returned SlowDown before body materialization, published no object, incremented only the exhausted rejection reason, and released pending reservations to zero; signed and presigned traffic, tamper rejection, exact ranges, graceful shutdown, and Compose multipart recovery also passed in S3 gateway run 34528570397

• packaged-image scratch-capacity CI passed at commit f8878c1584e: the non-root/read-only gateway reported positive total/free/available bytes and healthy zero-failure probe state from its 512 MiB tmpfs, then passed signed header and presigned traffic, tamper/wrong-secret/unsigned rejection, exact ranges, graceful termination, and Compose multipart recovery in S3 gateway run 34525303204

Deliberate remaining profile limits

The gateway remains the documented frozen S3 compatibility profile rather than every AWS S3 control-plane feature. Gateway-authored large payloads use verified Git LFS storage; existing Crab/Xet pointer objects retain chunk-aware range reconstruction and Xet deduplication. Moving S3 uploads themselves onto Xet would be a separate storage-format migration requiring atomic pointer publication, GC reachability, clone/hydration, and SDK compatibility proof.

Each PutObject, UploadPart, and copied source range still uses scratch proportional to that individual request body or range, with concurrency bounded by admission. Large multipart completion no longer uses scratch proportional to the assembled object: it validates frozen durable parts, then replays new content through verified LFS publication. This costs a second backend read for a new content hash; already verified LFS content skips the replay. A future single-pass design would need provider-portable content-addressed promotion with the same integrity, retry, GC, and restart guarantees.

Open uploads now have a persisted absolute expiry, registered-byte quota, distributed active-session bound, and a separate transient quota shared by in-flight transfers and retired replacement payloads. Every backend part write is preceded by a durable reservation; replacement registration moves the old payload into durable retirement in the same CAS that selects the winner. Terminal sessions retain their capacity slot until pending writers clean themselves, while the reconciler reclaims crashed transfers only after the persisted expiry plus a ten-minute provider-drain grace period. This bounds physical staging to twice the configured per-upload budget without leaving crash-created bytes unaccounted.

A Completing session is intentionally never expired because its frozen part set may already have published the object. New completions persist the planned result and bind their commit to an immutable publication-plan receipt, so reconciliation remains exact after later overwrites; current-object attributes retain compatibility for older records. An intent with no provable committed result remains fenced because absence of a receipt cannot prove rejection. Terminal lifecycle records and publication-plan evidence are retained for idempotent retries, so retention-safe compaction remains future lifecycle work.

The private management listener now exposes per-process request/admission saturation, aggregate multipart-reconciler health, lifecycle-owned content scratch, actual scratch-filesystem total/free/available capacity, pending reservations, retained headroom, and bounded capacity-rejection reasons with fixed labels. Logical backend latency, outcome, in-flight, and byte breakdowns are also exported with fixed labels. Provider wire-attempt telemetry, an explicit application-level Kubernetes policy cap, dashboards, and live alert delivery remain explicit production-observability work. The Helm chart now provides opt-in, release-scoped scrape and alert resources with 14 syntax- and semantics-tested rules. Cache-layer hit/miss/failure and verified-byte counters plus read-only catalog usage/health gauges are implemented with fixed labels; live Prometheus selection and Alertmanager receiver delivery remain unqualified. Kubelet remains authoritative when an emptyDir.sizeLimit is not exposed as a filesystem quota.

SlowDown under a bounded saturated queue is intentional S3 backpressure, not silent loss. Every gateway-generated response now includes Retry-After: 1; clients should still use standard exponential backoff with jitter, while operators size per-instance transfer capacity and scale replicas from admission pressure. High Git-pack counts are handled outside request acknowledgement: continuously supervise one crab metadb owner per repository for bounded geometric repack and monitor its candidate/action fields, with crab repack as the explicit catch-up command for an already fragmented repository.