feat(assignments): Change layout of the worker/portal assignments - #220
Merged
Merged
Conversation
A batch job that rewrites a chunk can't overwrite the ingested copy in place without breaking readers mid-run. It now writes under its own prefix and publishes by bumping the chunk's version: - Worker chunks carry `version`, and their dataset a `generations` list mapping each non-zero version to the prefix its files hang under, relative to `dataset_base_url`. Version 0 keeps the ingest layout (dataset_base_url + id); anything else inserts the generation's base_url between the two. `WorkerAssignment::chunk_url` resolves that. - Portal chunks carry `version` alone -- a portal never downloads, so it needs no prefix. The builder validates what the reader can't recover from: a chunk may not name a generation its dataset never registered, and version 0 may not have a prefix. Generation registrations are per-dataset, cleared by `finish_dataset`.
Two gaps in the versioning commit: - Version 0 is a normal version whose defining property is having no GenerationEntry, not a special-cased absence. Nothing behaved otherwise, but the schema and the builder's refusal to register a prefix for it read as though 0 were disallowed rather than simply entry-less. - `PortalAssignmentChunk::version()` was callable but the type had no public name, so a portal could read the version inline yet not write a function or struct over the chunk carrying it. `sqd_assignments::fb` now exports the generated views the readers already hand back.
The worker and portal schema comments narrated the split away from the combined assignment -- what the legacy format did, what was out of scope at the time, which field used to restate another. That history belongs in the log, not in a file describing a wire format.
Every field in both schemas now carries an explicit `id`, which is what fixes its vtable slot -- declaration order no longer does. Without that, adding `version` where it reads best renumbered every field after it; from here a new field takes the next free id and can be declared anywhere. This PR still breaks the format once (slots 8-12 of the worker chunk change meaning); the ids are what make it the last time. Dropping `dataset_id` from both chunk tables rides along, since it edits the same two tables. A chunk is only reachable through the dataset holding it, so the caller always has one already -- `find_chunk` and `find_chunk_by_timestamp` are handed the id outright, and iteration starts from the dataset. The one in-tree reader of the field was `chunk_url`, which used it to binary-search back to the dataset it had just come from. Measured on 100k chunks: 10,000,600 -> 9,200,632 bytes, 8 bytes per chunk, 8% of the blob. It costs 8 rather than 4 because the offset also forces alignment padding -- which is why removing a second dataset-level field saves nothing, and why `version` itself was free. `chunk_url` moves to the dataset that owns the generations; `WorkerAssignment::chunk_url` now takes a ChunkRef for the detached case. `iter_chunks_with_dataset`, `get_dataset_by_ref` and the `ChunkRef` accessors give a caller the dataset the chunk no longer names.
A portal reads whole columns and never a whole chunk, so a vector of chunk tables spends an offset, a vtable pointer and padding on every one of millions of chunks to no end. The dataset now carries the columns directly, all subscripted by the chunk index that bisecting first_blocks returns. 100k chunks, one dataset: 10,000,200 -> 3,401,848 bytes, 100.0 -> 34.0 per chunk. `id: string` goes. It was more than half the record, and the portal does nothing with it but copy it into Query.chunk_id, so it is reassembled from top, first_block, last_block and hash on read -- same string, same query protocol. The builder still takes it as one piece and splits it, so callers don't change and the two halves of the format can't drift. first_blocks are absolute rather than deltas because find_chunk bisects them, and prefix sums would make that linear; ts_offsets are absolute against base_timestamp for the same reason. block_deltas is new. A chunk's end was not on the wire at all, leaving the portal to infer it from the next chunk's start -- wrong wherever chunks aren't contiguous, and they legally aren't (archive.py asserts `self.next_block <= first_block`, non-strict). Those datasets had the portal claiming blocks that don't exist; find_chunk now answers InGap. A chunk's span is bounded by --chunk-size, so uint32 holds it. hashes are a struct column, flat with no per-element offset, NUL-padded rather than '0'-padded because '0' is a valid hash character and writers emit 5 to 8 of them. tops are runs, ~N/1000 entries: the top directory is constant across a directory and closed on a chunk count rather than a block boundary, so it can't be derived. 80 KB against 40 MB on 5M chunks. It is the one column the chunk index is searched for rather than subscripted into, so the builder enforces what makes that search sound: first run at index 0, strictly ascending. versions are dense. Runs would be 8 bytes against 4 and only pay while one covers more than two chunks, which fragmented backfills break; dense has a predictable ceiling and the column is absent entirely until something is backfilled. worker_offsets/worker_indexes are CSR -- how a column of variable-length lists is expressed at all, the alternative being a vector of wrapper tables. No RLE over it yet: whether that pays depends on whether a worker's chunks form contiguous slices, which wants measuring for real.
Removing it as redundant was wrong: it is the head block's full hash, while the `hashes` column holds the truncated short hashes that go into chunk ids. Different values, and the full one is not recoverable from the short one. It takes id 12 rather than slotting in after `last_block` where it is declared -- which is what the explicit ids are for, and renumbering to tidy it would contradict the rule the file states. Costs 80 bytes on a 100k-chunk dataset: 3,401,848 -> 3,401,928, still 34.0 bytes per chunk.
Cut them down, keep them in the present tense, and say what each field is rather than what it isn't. Drops the Rust snippet from TopRun -- these schemas are read by more than one language, so the invariant is stated in prose instead. The worker schema gets the same pass, since the two share a header and would otherwise drift apart in style.
Every chunk of a dataset hangs off the same base url, so the dataset holds it and a chunk names only what distinguishes it. The chunk builder still takes it — call sites are unchanged — and now rejects a second chunk of the same dataset naming a different url, which the old shape could only express by silently disagreeing with itself. This buys no space: 9,200,632 -> 9,200,592 bytes on 100k chunks, still 92.0 per chunk. Removing `dataset_id` earlier freed 4 bytes of alignment padding along with the offset, and this offset falls straight into that padding. The chunk table would have to shed 8 bytes at once to shrink again. Deleting a field mid-table forces the ids after it down one, since flatc requires them consecutive from 0. Renumbering is safe here only because it rides the same unpublished break as the rest of the PR; once anything reads these blobs, retiring a field means `(deprecated)`.
The portal format wants timestamps narrower than the legacy `uint64` of absolute epoch milliseconds. This measures what each candidate encoding would actually cost on a real assignment, rather than guessing. On mainnet (6,358,494 chunks, all timestamped): milliseconds offset from a per-dataset base overflow `uint32` for 94.1% of chunks and 241 of 245 datasets, since `uint32` milliseconds span only 49.7 days. Seconds overflow nothing until 2106, and 92.1% of chunks are whole seconds already -- the 7.9% that aren't sit in 81 datasets, three quarters of them in `hyperliquid-replica-cmds` alone. So the choice is 8 bytes for milliseconds against 4 for seconds and losing the sub-second ordering of one chain that has it. The module header says how to run it and how to read the output.
One question, one answer: which chunks carry a millisecond timestamp, meaning one too large for a uint32. Prints `s3://dataset/chunk_id <timestamp>` and nothing else; the base, delta and distribution analysis it grew during the encoding discussion have served their purpose. Output is a buffered locked stdout, which also stops `| head` from panicking on the broken pipe.
The interesting defect is a timestamp written in seconds where milliseconds were meant, so the report now lists chunks whose value fits a uint32 rather than those that exceed it. Milliseconds have been above u32::MAX since 1970-02-19 and seconds stay below it until 2106, so the width of the value is what separates them. On mainnet this finds 3030 chunks, and every one of them holds exactly 0 -- missing timestamps rather than mistaken units. 3024 are in solana-mainnet-3, 5 in asset-hub-polkadot-4, 1 in shiden-substrate-4.
Carry timestamps as absolute milliseconds, and add a converter that turns a legacy assignment into a worker and a portal one. The portal's `base_timestamp` + uint32 offsets could not hold real data: uint32 milliseconds span 49.7 days, and 241 of mainnet's 245 datasets carry more history than that, so 94.1% of chunks had no encoding. Seconds would have fit in 32 bits, but 7.9% of chunks carry sub-second values -- and in `hyperliquid-replica-cmds`, where consecutive chunks share a second, truncating would make find_chunk_by_timestamp unable to tell them apart. So `timestamps: [uint64]` holds the value as ingest records it, and the base goes away with the arithmetic that needed it. The converter derives what legacy does not carry. Schemas come from the file lists: a table is a `*.parquet` name minus the extension, a dataset's roster is the union of its chunks' tables, and a chunk missing tables gets a bitmap. Chunk ids stay whole on the worker side and split into columns on the portal side. Sealed headers are copied byte for byte, since the Cloudflare secret that would mint fresh ones never travels with a blob -- which `add_worker_with_sealed_headers` now makes possible, filling a hole in a builder that could previously only mint what it emitted. Timestamp anomalies are reported, never repaired: mainnet has 3030 chunks whose timestamp was never recorded and 6 that step backwards, both of which the legacy format has and its own reader comments on. Repairing them here would fabricate data and hide an ingest defect. Every conversion is verified against its source -- chunk ids, block ranges, timestamps, versions, worker indexes, resolved tables, dataset order and heads, worker identities and header bytes -- because nothing else checks 6.36M chunks. On mainnet: 1,168,393,408 bytes of legacy become 654,637,120 of worker (56.0%) and 290,278,032 of portal (24.8%), or 260 MB and 153 MB gzipped.
`--compress gzip|zstd|both|none` picks which compressed copies land beside the plain `.fb`, both by default, and `--zstd-level` sets the level. zstd defaults to 9 rather than its own 3. On the mainnet portal blob, gzip gives 152,981,219 bytes in 9.6s; zstd -3 gives 154,181,827 in 0.6s -- faster but larger, which makes the option pointless -- while -9 gives 149,400,515 in 2.6s, beating gzip on both. -19 reaches 136,838,851 but takes 63s, which is the wrong trade for a blob that is compressed once and downloaded by everyone.
The size table now carries the time beside each compressed size, so the two halves of the trade
are read together rather than one from the table and the other from the progress lines above it:
worker 654637120 259992345 (18.3s) 251514121 (7.7s) (56.0% of legacy)
portal 290278032 152981219 (9.7s) 149400515 (2.7s) (24.8% of legacy)
The zstd column is labelled with the level in use, since the number means little without it.
It answered the question it was written for -- how timestamps could be carried in the portal columns, and which chunks carry values that aren't milliseconds -- and that answer is now in the schema. Keeping a one-off diagnostic in the tree means maintaining it against a format that is still moving.
Handed a .gz the tool parsed it as flatbuffer bytes and failed with "Type `i32` at position 559903 is unaligned", which says nothing about the actual problem. It writes .gz and .zst, so it now reads them too, detected by magic bytes. That exposed the size report comparing outputs against the input *file*, which for compressed input is the compressed size -- the worker blob came out as "195.5% of legacy". It now compares against the uncompressed source, and prints the source's dataset, worker and chunk counts, since the ratios only mean something next to the shape they came from.
Measures both along the access patterns the portal and worker actually use, since benchmarking either as a plain scan would flatter it. A portal walks chunks in order but keeps no cursor, so every step repeats a string-keyed dataset search and a chunk search; a worker scans its own chunks once per applied assignment and then does nothing but ChunkRef dereferences. Both formats are measured doing the same job rather than the same calls -- a stream step needs the chunk's end block, which legacy parses out of the id and the portal reads from a column. The fixture is synthetic so the benchmark stands alone, but shaped like mainnet: 200 datasets, 2M chunks, 7 replicas each, 2000 workers, head hash on the last chunk of each dataset only.
The worker was still a flatbuffer table per chunk, which cost 28 bytes of header, vtable pointer and vector slot before any content, and padded every uint32 to 8 bytes besides. Measured on mainnet, that structure plus the chunk id string was 84 of its 103 bytes per chunk -- so dropping files, dataset_id, base_url and the rest had only ever attacked the remaining 19, and the format sat at 86% of legacy. Columns instead, mirroring the portal: 1,026,228,328 -> 421,753,120 bytes, 103.3 -> 42.5 per chunk, 86.1% -> 35.4% of legacy. The chunk id goes the same way it did for the portal -- rebuilt from tops, first_blocks, block_deltas and hashes rather than stored, which is 40 bytes a chunk against 12. `ChunkHash` and `TopRun` move to assignment.fbs, since both formats now describe chunk identity the same way and should not describe it twice. write_schema_id travels as runs like tops, not as a dense column or a dataset-level default. A schema holds for a whole dataset until it evolves and then holds again, so runs cost nothing where dense would cost 4 bytes a chunk -- the opposite of `versions`, where backfills fragment the runs and dense wins. tables_present becomes a fixed-stride bitmap column, wide enough for the widest roster the dataset uses, absent entirely where no chunk trims its tables. Verification of the whole blob drops from 742 ms to 470 us -- 1578x -- because a verifier walks a handful of vectors per dataset instead of ten million tables. Against that, a per-chunk scan costs 6.5 ms more per applied assignment and building a download url costs 142 ns more, since the id is assembled rather than copied. Both are paid once per apply or once per download. One test goes: ordering `tables_present` before `write_schema_id` used to encode a bitmap against the wrong roster, which `finish` had to catch. Columns resolve the names against the final schema, so the hazard no longer exists and the test now asserts that the later id wins.
Columnarising them was a redesign that wasn't asked for. write_schema_id goes back to one value per chunk -- a dense `write_schema_ids` column rather than runs -- so a chunk names its schema outright and no invariant governs how they may change along a dataset. tables_present goes back to a per-chunk bitmap that is present or absent, with `tables_present` fallible again, validated against the roster the moment it is called, and `finish` rejecting a `write_schema_id` set after it. The test covering that ordering hazard comes back with it. Variable-length per-chunk bitmaps travel as CSR, the same shape `worker_indexes` uses: chunk i's bits run from tables_present_offsets[i] to [i+1], an empty slice means every table is present, and both columns are absent where no chunk in the dataset trims. Identical bitmaps are no longer shared -- CSR offsets ascend, so a slice cannot point backwards -- and nothing is lost, since the old interning saved a 4-byte pointer per chunk while the bits themselves are narrower than that. Costs 43 MB against the run encoding: 421,753,120 -> 464,888,880, 35.4% -> 39.0% of legacy.
`SQD_BENCH_LEGACY=<path>` benchmarks that assignment and the split pair the converter wrote beside it, instead of the synthetic fixture. The legacy input may be plain, gzipped or zstd-compressed. Probes are drawn from real datasets at real chunk boundaries and kept only where both formats resolve them, so neither side is timed doing less work than the other. The load benchmark copies its blob once per iteration rather than in batches, since a batch of 1.2 GB clones is not a batch. Mainnet is a harder test than the synthetic stand-in and moves several numbers: verification 1390x -> 11200x, the per-chunk scan 69% -> 72% slower, and get_worker's binary search 8x faster on both formats, since 1919 real peer ids compare on their first bytes where generated ones do not.
Two hot paths resolved a column through the vtable on every element they touched, which is a lookup and an Option per step for data that cannot change during the walk. `find_chunk_by_timestamp` re-read `timestamps()` inside the bisect, so ~23 probes over a 50k-chunk dataset did 23 vtable lookups to reach the same vector. Hoisting it out and branching once on the column's presence: 8.11us -> 4.19us on mainnet, which turns a 21% loss against legacy into a 38% win. `iter_chunks_with_ref` reached the routing columns through each chunk, so a scan of ten million chunks built ten million iterators over two vtable lookups apiece, to find the thirty-odd thousand belonging to one worker. Resolving them once per dataset instead: 86.3ms -> 56.1ms, a 72% loss against legacy down to 12%. What remains is inherent to CSR: two offset reads per chunk where legacy dereferenced one vector. The scan is still O(all chunks) to find one worker's own, in both formats.
Rebuilding an id from columns looked like the cost of dropping the string, but it wasn't: the
three `{:010}` values went through `Formatter::pad_integral` at about 35ns each, and `url()` then
allocated a second time to concatenate. Measured in isolation: copying a stored id 74ns, rebuilding
as we did 192ns, rebuilding into one allocation 139ns, rebuilding with the digits written by hand
34ns -- faster than the copy, since that pays std::fmt and an allocation too.
So the padding is written directly and a url is assembled in one reserved String. On mainnet,
download_url 14.81us -> 5.85us per 64, which is 211% slower than legacy down to 13%; portal
chunk_id 15.70us -> 10.07us, 129% down to 47%. What remains there is a memcpy against a rebuild,
which is the trade the columns are for.
Values too wide for the pad still print in full, as `{:010}` does. All 9,933,060 mainnet chunks
still reassemble byte-identically.
`--out-dir` keeps the default -- the directory the tool was run from -- but means the documented workflow can run from a source tree without dropping 900MB of blobs into it.
Found reviewing against the Rust guidelines, with a test that proves it: a blob could pass `from_owned` -- the verified door, which is what a worker applies through -- and then panic inside flatbuffers on the first chunk read. The verifier checks each vector on its own; nothing in the encoding says the columns must agree in length, and it is precisely their agreeing that lets one chunk index subscript all of them. A dataset claiming three chunks with one entry in `block_deltas` verified fine and blew up on chunk 1. The legacy format had no such gap, since a chunk table carried its own fields and had nothing to disagree with. So `from_owned` now checks, for every dataset, that the dense columns match the chunk count, that the CSR offset columns are one longer, and that the run columns cover chunk 0. It costs 15us on mainnet's 204 datasets -- load goes 334us to 349us against the 3.67s legacy takes -- because every check is O(datasets) or O(runs). Invariants that would cost O(chunks) are left out: none of them can panic, since the slice accessors clamp, and paying milliseconds to catch them would undo the point of a format that loads in microseconds. That needs an error type the crate owns, since no `InvalidFlatbuffer` variant can say "these columns disagree" -- `InvalidAssignment` wraps the flatbuffers error and adds one variant naming the dataset and the column. `from_owned_unchecked` is unchanged and now documents that it can panic where the checked door returns. Also from the same review: every builder setter is `#[must_use]`, since they return `Self` and a chunk is only staged by `finish` -- `builder.new_chunk().id(..)` on its own silently staged nothing and now warns. `# Panics` sections where a public method can panic, `id()` reserving what an id needs rather than what a url needs, and `partition_point` taking `FnMut`.
The two formats share the checker but not the column list, and a wrong field name in either list would check nothing while passing everything. The worker had a rejection test; the portal only had the real 460MB blob proving it raises no false positives.
Both formats rebuild a chunk id by writing the zero-padding directly, because `{:010}` costs about
35ns a value. That only holds up if it agrees with `{:010}` exactly, and mainnet cannot check the
interesting half: the largest block number in the assignment is around 70 million, so nothing
there is wide enough to overflow a ten-digit pad -- which is precisely where writing digits by
hand goes wrong.
So the two are compared directly at 0, at the pad boundary either side, and at u64::MAX.
Checked against the real assignment separately: all 9,933,060 reconstructed ids equal the legacy
ones position by position, in both formats, and the id sets are equal. The 633 ids that appear
twice are two copies of one chain -- ethereum-mainnet-tb and ethereum-mainnet-4 share block ranges
and hashes -- so a chunk id is unique within a dataset, not across the assignment.
Neither split format keeps `dataset_id` on the chunk -- a chunk belongs to the dataset holding it -- so dropping the field is lossless only while the two agree. That holds for all 9,933,060 mainnet chunks, but it is the source's to break rather than ours to assume, so the converter now checks it per chunk instead of trusting it. Verified separately against the real assignment: dataset+id pairs match legacy position by position and as sets, in both formats, and all 9,933,060 pairs are distinct -- unlike the id alone, which repeats 633 times where ethereum-mainnet-tb and ethereum-mainnet-4 carry the same chunks.
The worker builder generated a keypair at construction and wrote its public half into the buffer as `common_identity`, whether or not anything sealed headers with it. An assignment converted from another one copies its headers verbatim -- the Cloudflare secret never travels with a blob -- so those 32 bytes sat unreferenced, and being random, made every build of the same input differ. Found while chasing why gzip of two supposedly identical conversions differed by one byte: the blobs were not identical. The identity is now written on first use, so a conversion is reproducible -- same input, same bytes, same hash -- which matters for anything publishing these by content hash. Mainnet's worker blob loses exactly those 32 bytes: 464,888,880 -> 464,888,848.
A chunk id is parsed by fixed offsets at the portal, not by pattern: separators at 10, 21 and 32,
and a total length between 38 and 41 bytes, with the hash landing in a zero-padded [u8; 8]. Since
the id here is derived rather than stored -- three ten-digit numbers, three separators, then the
hash -- a hash under five characters builds a 34 to 37 byte id that the portal rejects at runtime.
The builder accepted 1 to 8 and so could emit ids no portal would read.
It now takes 5 to 8, the same range worker-rs matches with `\\w{5,8}`, and a test pins the derived
string to the portal's offsets so the two cannot drift apart silently. Nothing changes for real
data: mainnet's hashes are 5 or 8 characters, and both blobs convert to the same bytes as before.
A pass over every comment in the crate: deleted the ones restating the item they sat on, trimmed the rest. 536 comment lines to 449, and the ones left say something the code doesn't. Two were also wrong. The worker sections of reader.rs and builder.rs still described chunks as tables mirroring the legacy layout, from before they became columns, and a test still claimed two chunks with the same tables share one bitmap -- true when bitmaps were interned, not since CSR offsets made a slice unable to point backwards.
define-null
commented
Aug 17, 2026
`cargo +nightly fmt --all -- --check` is a CI step, and the dataset's finish call landed at 103 columns against rustfmt.toml's max_width of 100. Nightly is required for the check to reproduce: rustfmt.toml sets imports_granularity, which stable silently ignores. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Make zstd a regular dependency It only had a dev-dependency entry, which covers the bench and the convert_assignment example but not library code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Stop allocating per staged chunk in the split builders Staging a chunk allocated up to three times: the chunk id String, the worker-index Vec, and the tables bitmap on partial chunks. None outlive finish(). - id() now parses the id instead of copying it; the Result is held so a malformed id still surfaces at finish(). Everything staged from it is u64 or ChunkHash. - The worker indexes and the tables bitmap move to buffers on the parent builder, cleared by new_chunk(), so they amortize to zero allocations. Staging 1M chunks in release: worker 117 -> 95 ms, portal 80 -> 72 ms. No API change. The range-mismatch error no longer quotes the raw id, only the block numbers it disagrees on; malformed-id errors still carry the whole id. A chunk that never names workers would inherit the previous chunk's list if a buffer went uncleared, which nothing covered, so both worker-slice tests now include such a chunk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop two comments that only restated the destructuring Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Defnull <879658+define-null@users.noreply.github.com> Reviewed-on: http://localhost:3000/defnull/sqd-network/pulls/5 Co-authored-by: claude <claude@example.com> Co-committed-by: claude <claude@example.com>
The converter writes .gz and .zst, so reading them back by name keeps the two ends on one convention. A mislabelled file now fails as a bad gzip/zstd stream, or as a bad flatbuffer if it claims to be plain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this PR about?
Chunks need a version. A batch job that rewrites a chunk cannot overwrite the ingested copy in place without breaking readers mid-run. So a chunk carries a
version: 0 is the ingested copy, any other value names aGenerationEntrywhose prefix its files hang under. A job writes to its own prefix and publishes by bumping the version, and abandons a bad generation by never publishing it.The id cannot carry that. Identity travels as
<top>/<first>-<last>-<hash>, parsed separately by four components, so extending a chunk means smuggling a field into that string and hoping they agree. worker-rs already tried, adding a trailing suffix so several chunks can cover one block range —(\d{10})/(\d{10})-(\d{10})-(\w{5,8}).*, under a// TODO: synchronize with other language implementationsnobody took up:.*captured into the idsplit('-')thenget(2)Displayprints without it^(\d+)-(\d+)-(\w+)$One of four honours it, and the silent failures are the disqualifying ones:
sqd-networkloses the suffix on round-trip, collapsing two versions of a chunk into one — exactly the distinction a version exists to preserve. So it is a field, and the id stops being an extension point: the parts live in columns and the canonical string is derived, so the next fact a chunk needs is a column rather than a suffix one parser understands.Both formats are still too large. Splitting them barely shrank them: the worker assignment as merged is 1,026 MB against legacy's 1,191 MB, because both sides kept a flatbuffer table per chunk — 28 bytes of header and vector slot before any content — plus the 40-byte id string. Dropping the fields each side didn't read only attacked the remaining 19 bytes of 103. Chunks are columns now, and with the id derived rather than stored, both formats land near 39%.
What changed
version+generationson the worker side; the portal gets the version without the storage prefix, since it never downloads.block_deltasis new. A chunk's end was never on the wire, so a portal inferred it from the next chunk's start — wrong wherever chunks aren't contiguous, and they legally aren't.find_chunknow answersInGap.Numbers
Real mainnet: 204 datasets, 1,919 workers, 9,933,060 chunks.
gzip and zstd tie on size at these levels; zstd is ~3.5× faster. Both reach ~46% at
zstd -19, for ~85 s.An applied assignment goes from 3.79 s to 56 ms. What is slower is rebuilding an id from columns rather than copying a stored string — paid once per S3 fetch, against 3.7 s of verification that no longer happens.
Two dataset-level microbenchmarks are left out because they don't reproduce:
scan_datasetshas measured 1.11, 2.35, 2.21 and 2.19 µs across four runs on a byte-identical blob with nothing on that path changing. A few microseconds over 204 datasets is small enough that allocation addresses dominate, so neither it norget_datasetsays anything reliable at this granularity.Reproducing it
From the repo root, with the legacy assignment at
/tmp/mainnet.fb.1.gz(plain, gzip or zstd):cargo run --release -p sqd-assignments --all-features --example convert_assignment -- \ /tmp/mainnet.fb.1.gz --out-dir /tmp SQD_BENCH_LEGACY=/tmp/mainnet.fb.1.gz \ cargo bench -p sqd-assignments --all-features --bench assignmentThe converter ends with
verified: both assignments reproduce the source, which appears only after all 9.9M chunks have been checked against the legacy blob — ids, block ranges, timestamps, versions, worker indexes, resolved tables, dataset heads and every worker's sealed headers.--verify-onlyre-checks existing outputs;--compress noneskips the compressors.Worth a look