Repository navigation
perf(engine): copy export chunks once and hash key groups in one pass - #1347
Conversation
- Export's ordered cursor interleaves a chunk's rows straight from the hydrated batches (`interleave_record_batch`) instead of concatenating them and then taking: one copy of the chunk instead of two. The key-order check compares borrowed ids instead of allocating two Strings per row. - Hydration takes each fetched row's bytes once, at fetch time, with the one-pass `row_bytes`, and an interleave's estimate sums them per pick instead of slicing every column of every pick on each halving. The estimate is unchanged. - The key-set block and the bound-edge source grouping number keys with DataFusion's `create_hashes` into a `HashTable` (`KeyGroups`), the pattern the hash join's build side uses, instead of a SipHash map of string slices, one row at a time. Closes ModernRelay#1340
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1347 at 1982ff74
Outcome: correct; one small evidence item before merge. I found no wrong result and no unbounded memory path.
What and why
This PR removes three CPU and copy costs from #1340. Export builds each ordered chunk with one interleave_record_batch instead of concat_batches and then take_record_batch. Hydration computes each fetched row's byte size once, not once for each estimate. The key-set block and the bound-edge source grouping hash their keys in one vectorized pass into a hashbrown::HashTable, the same pattern hash_join uses. Rows, order, and admission rules do not change.
Correctness (checked)
- Hashing.
create_hasheshashes a Utf8 array throughvalue(i). It obeys the slice offset and does not write null rows, whichKeyGroupsskips (datafusion-common54.0.0hash_utils.rs:984,:1100, first arrayrehash = false). The sliced-probe assertion in the new test also covers this. - Collisions. The table compares the stored hash and then the string through the first row. Two keys with one hash stay separate groups.
- Estimate.
interleave_estimatenow sumsrow_bytesper pick plus16 × columns, then adds128 × columnsand the scratch from the first batch. This is the old per-column sum regrouped.interleave_estimate_from_row_bytes_equals_the_per_slice_sum_issue_1340pins it, androw_bytes_equals_the_per_row_slice_for_every_layoutalready pinsrow_bytes=slice_bytes(row, 1). - Export. In arrow-select 58.4.1,
interleavehas native paths for Utf8/Binary/views, Struct and List, and usesMutableArrayDatafor other types (interleave.rs:98-111). Blob descriptor structs and vectors are therefore covered. The wide-row and Blob export owners pass. - Bound edges. The CSR keeps each group's rows in input order, so output order inside an edge row does not change. The key list now has first-appearance order instead of
HashMaporder. It only feeds thekey IN (...)filter (graph.rs:1877), so order has no effect there. - Memory. The new charges are above the real sizes: about 75 B per source row against 236 B charged (table entry, first row, row group, CSR, owned key).
Blocking
B1. A cost claim with no checked-in instrument. The changelog says correlated blocks, bound edges and hydration "spend less CPU". The PR body has timings from a scratch benchmark that is not in the tree, and (2) was not timed. docs/dev/invariants.md #13 says "Cost claims require a checked-in instrument", and testing.md rule 5 puts time measurements in instruments. #1342 added integration/scan_limit_cost.rs for the same reason. This is a small fix. Choose one:
- Check in the scratch benchmark as
#[ignore = "instrument: …"]tests, for example inintegration/anti_join_cost.rsfor (3), which already measures the key-set strategy, and a sibling for the export chunk (1). - Or reduce the changelog to what code inspection proves: one ordered-chunk copy instead of two, with no CPU claim.
Optional
O1. One owner for the string-keyed table. This PR adds KeyGroups, but hash_join.rs:155-184 (index_rows, find_row, hash_ids) is the same structure: one (hash, u32) entry for each distinct non-null Utf8 value, compared through the array, with the same HASH_SEED. KeyGroups::firsts[group] is the build row that hash_join stores. Open #1345 adds a third copy (TypeIndex::dense_of, a HashTable<u32> with its own fixed-seed ahash). Null handling, collision comparison and seed choice now have three places to drift. Suggestion: let KeyGroups own its StringArray (an Arc clone) so hash_join::Built can hold one, and delete the three helpers. Then #1345 can use it or state why it needs a different hasher. Without this, five more changes like this one give five tables.
O2. Table capacity. KeyGroups::build starts from HashTable::new() and rehashes as it grows. index_rows uses with_capacity(ids.len()). KeyGroups::charge already admits one entry per row, so with_capacity(keys.len()) adds no admission risk and removes the rehash work from the hot path this PR targets.
O3. Uncharged probe buffers (PR text). The PR says the probe's 8 B hash per inner row is uncharged. The uncharged probe state is actually 16 B per row: the u64 hash and the Option<u32> group (anti_join.rs hashes/inner_groups, graph.rs probe_hashes/probe_groups), plus the existing 4 B tags. Each buffer is bounded by one input batch, so this is acceptable. Please correct the sentence, or charge the buffers to the batch's input_memory.
O4. Null source ids. rows_by_src used to key a null source id as "". Now the row is dropped without a message. I agree that no supported plan makes a null <var>.@id. If one ever appears, it is an integrity fault, and invariant 8 asks for a loud failure. A src_ids.null_count() > 0 check that returns manifest_internal would make that hold by construction. The key-set block can keep treating a null key as "no matches", because that is its documented contract.
Note (no action). Two loops lost their per-step cancellation check: the source-grouping loop in graph.rs and the per-pick loop in interleave_estimate. Both are now simple linear passes over one held batch, and each operator checks before and after, so cancellation latency does not change in a way that matters.
Liability
The PR removes a std HashMap with SipHash at two sites, an extra chunk copy, and repeated per-column to_data() calls. It adds one new type (KeyGroups, 178 lines with its test) and one new caller obligation: interleave_bytes / interleave_once take row_bytes from the caller. Hydration is the only caller, and the test pins the result. The net liability is small and goes down if O1 is done. No API, persisted format, Lance surface, or storage-cost golden changes. The DST golden is unchanged, and CI's DST pinned suite passed.
Validation
- Local, exact head
1982ff74(isolated worktree,cargo dev-nextest):- In-source:
key_groups_number_keys_like_a_string_map_issue_1340,interleave_estimate_from_row_bytes_equals_the_per_slice_sum_issue_1340,row_bytes_equals_the_per_row_slice_for_every_layout,a_take_admits_a_copy_per_index_and_a_null_row_per_null_index, the hydrate fragment test, and the tworeport::testsanti-join tests: all passed. - Integration
export::,engine_v2_memory::,changes::,merge_fast_forward::,traversal_indexed::,engine_v2::,aggregation::: 161 tests, all passed. One test (changes::history_reads_are_the_same_after_a_merged_branch_is_deleted_and_recreated) overflowed the default thread stack on this host and passed withRUST_MIN_STACK=16777216, whichci.ymlsets.
- In-source:
- CI on the exact head: Test Workspace, GQ Logic Tests and GQT (ordinary, served, dst), clippy, fmt, DST pinned suite, RustFS S3 (default and failpoints), Azurite, Storage Upgrade Compatibility and Fix Regression Gate passed (28 checks; 4 tag/release jobs skipped).
- Regression test: not applicable. This is a cost change with identical results, not a defect. GQT cannot show it, and the PR adds the right mechanism-level equivalence tests.
- Other open PRs:
git merge-treeof this head with #1341 (anti_join.rs) and #1345 (graph.rs) is clean. - Not done: no timing on this host, so I did not verify the CPU figures. I did not run the GQT corpus, S3 or Azure locally (CI covers them). The
anti_join_costinstrument was not run.
…dd cost instruments KeyGroups now owns its key array and replaces the hash join's own build table (index_rows, find_row, hash_ids), so the hash join, the key-set block and the bound-edge source grouping share one string-keyed table, seed and collision rule. Its table is sized for every row, which its charge already admits, so it never rehashes. The bound-edge source grouping refuses a null source id as an integrity fault instead of dropping the row. Each CPU claim has an ignored instrument test that times the old code against the new on the same input and asserts equal results: the export ordered chunk, the hydration estimate and the key groups.
|
Addressed at B1. Each cost claim now has a checked-in
Release build, best of 30 (chunk, estimate) or 10 (key groups), on the pinned toolchain. The commands are in the PR body. I put them in source rather than in O1. O2. The table and the first-row list are created with capacity for every row, which O3. I corrected the PR text: the probe state left out of the memory charge is 16 B per inner row (hash plus O4. Checked locally: fmt, |
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1347 at e7e22cd4 (delta since 1982ff74)
Outcome: ready to merge. No blocking findings. The earlier review of 1982ff74 asked for one blocking item (B1) and four optional items (O1 to O4). be615840 and e7e22cd4 address all five. I found no new defect. I have one small optional item (O5).
What changed since 1982ff74, and why
- B1, cost instruments. Three
#[ignore = "instrument: …"]tests now time the old code against the new code on the same input. Each one asserts that both give the same result:ordered_rows_against_concat_then_take_cost,interleave_estimate_from_row_bytes_costandkey_groups_against_a_string_map_cost. The old code stays only in the test modules, as a baseline and an oracle. The changelog now names what each path does differently. It no longer makes a general claim of "less CPU". - O1, one owner for the string-keyed table.
KeyGroupsnow owns anArcclone of itsStringArray. The hash join'sBuiltholds one.index_rows,find_row,hash_idsand the separateHASH_SEEDinhash_join.rsare deleted.HASH_SEED, null skipping and collision comparison now exist only inkey_groups.rs. I searchedcrates/forcreate_hashes,HashTable<andHASH_SEED: each occurs only inkey_groups.rs. NoHashMap<&str, _>is left inengine/operatorsorengine/graph.rsoutside tests. - O2. The table and
firstsare created with capacity for every row.group_bytesalready charges that capacity. - O3. The PR text now says that the uncharged probe state is 16 B per inner row.
- O4.
produce_bound_edge_pairsrefuses a null source id withmanifest_internalbefore it groups anything (graph.rs:936-942).bound_pairs_refuse_a_null_source_idchecks the refusal and that nothing is emitted.
Correctness (checked)
- Hash join probe. The probe hashes with
built.ids.hash(…), so it uses the build'sRandomState. It then mapsgroup_oftofirst_row. This gives the same build row as the oldfind_row: the first row of a duplicate id, and two ids with one hash kept apart by their strings. The forced-collision test moved unchanged tokey_groups_keep_the_first_row_and_resolve_a_collision, and it now also checks the per-row groups. - Build charge.
work.grow(KeyGroups::group_bytes(n))is the same as the oldentries::<(u64, u32)>(n)plusentries::<u32>(n)forfirsts(68 + 20 = 88 B per row). The hash buffer is still charged during the build and released after it (shrink_to(before + held)).firstsis real memory (4 B per row), so the larger charge is correct under theentriesconvention. buildandbuild_numbered. Both go through onehashed→from_hashespath. The only difference is the per-row callback. The test asserts that both give the samefirsts.- Null source refusal.
null_count()respects the slice offset. The check runs aftersource_memory.hold(wide)but before the grouping charges and beforeKeyGroups::build_numbered.source_memoryis a child lease, and itsDropreleases its share on the early return. Therefore a refused batch emits no pair and keeps no reservation. The key-set block keeps its documented contract: a null key means "no matches".
Optional
O5. Index the new instruments in the test-owner table. docs/dev/testing.md "Cost and benchmark contracts" lists every cost instrument. These are the first in-source instrument: tests in crates/omnigraph, and the row does not name them. One clause, for example "the in-source operator instruments in key_groups.rs, memory.rs (row_bytes_tests) and ordered_cursor.rs (ordered_rows_tests)", keeps the table the one place to find them. The planner's in-source derivation instrument is listed this way in the plan-acceptance row.
Note (no action). The instrument baselines are copies of the deleted code. They cannot drift in a harmful way, because each one is only a frozen baseline and an equality oracle. Production code does not call them.
Liability
This round removes more than it adds. It deletes the hash join's own table (three helpers, a seed constant and a test) and moves its single test to the owner. The PR now has one string-keyed table in the operators instead of two. The one added obligation is small: callers use build or build_numbered, and the probe uses hash then group_of. A sixth similar change would extend KeyGroups and would not add a table.
The cost is a larger hash-join build admission: 88 B instead of 68 B per build row. A build that was admitted close to the query budget can now be refused. The extra 20 B pays for a real 4 B firsts entry under the entries convention, so this is an honest charge, not a regression to fix. The null refusal turns a silent row drop into a loud internal error. This agrees with invariant 8. No supported GQ plan produces a null bound @id. GQT (ordinary, served, dst) passes on this head.
Open #1345 (TypeIndex::dense_of) can now use KeyGroups or say why it needs a different hasher. That decision belongs to #1345, not to this PR.
Validation
Local, exact head e7e22cd4 (clean, isolated worktree; cargo dev-nextest; RUST_MIN_STACK=16777216 as in ci.yml):
-
192 passed, 0 failed. In-source: all 32
omnigraph-enginelib tests that match key groups, hash join, anti-join, row bytes, take estimate, hydrate, expand stream, ordered rows andreport::tests. These include the five new or moved tests. Integration:export::(13),engine_v2_memory::(31),changes::(60),merge_fast_forward::(20),traversal_indexed::(14),engine_v2::(14) andaggregation::(8). -
The three instruments pass in a debug build (
--run-ignored only), so their equality assertions hold there too. -
Instrument timings, release build. I used the PR's command (
cargo test --release --locked -p omnigraph-engine --lib -- --ignored --nocapture …) on toolchain 1.97.1, on a Modal gVisor box (x86_64, shared CPU). There were two runs, and the three tests ran at the same time in one test process:Instrument old new ordered chunk, 8 × 1,024 wide rows 4,802 / 4,717 µs 1,395 / 874 µs hydration estimate, 1,024 rows, 2,048 picks 3,047 / 2,613 µs per estimate 1,439 / 1,391 µs once per window, then 4 / 4 µs per estimate key groups, 100k outer keys, 1M probes 55.1 / 53.6 ns/row 34.7 / 33.7 ns/row The direction and the approximate size agree with the PR body on all three paths. The absolute figures are higher on this host. On this host, one sizing pass (about 1.4 ms) costs about half of one old estimate (about 2.6 to 3.0 ms). Thus the sizing pass costs less than the two estimates that every window makes. These are micro-benchmarks on a noisy shared host, not end-to-end measurements.
Regression proof for O4. GQT cannot express this contract: no GQ query can put a null @id into a bound frontier. The cheapest Rust owner is expand_stream/tests.rs, and the PR uses it.
- Before the fix: I removed only the 7-line null check in
graph.rsate7e22cd4. At1982ff74the same function differs from that tree only by thebuild→build_numberedrename.bound_pairs_refuse_a_null_source_idthen failed for the predicted reason:called `Result::unwrap_err()` on an `Ok` value: (). The null row was dropped and the call succeeded. - At the exact head (edit restored,
git statusclean): the test passed. - Command:
cargo dev-nextest -E 'package(omnigraph-engine) and kind(lib) and test(bound_pairs_refuse_a_null_source_id)'.
CI on the exact head e7e22cd4: all 28 checks pass, including Test Workspace, GQ Logic Tests, GQT (ordinary, served, dst, dst-clippy), clippy, rustfmt, DST pinned suite, RustFS S3 (default and failpoints), Azurite, Storage Upgrade Compatibility and Fix Regression Gate. The 4 tag and release jobs are skipped.
Other open PRs. git merge-tree of this head is clean with current main (825213be) and with #1316, #1318, #1334, #1341, #1345, #1346, #1348, #1349 and #1357. These include every open PR that touches hash_join, anti_join, graph.rs, memory.rs or the anti-join GQT cases.
Not done. I did not run the GQT corpus, S3 or Azure locally; CI covers them on this head. I did not run clippy or fmt locally; CI passed both. I timed no end-to-end export, hydration or join query. Thus the cost evidence is the micro-benchmark instruments only, as the PR body says.
Summary
Three small hot-path costs from #1340, each replaced by an Arrow kernel or the repository's own hashing pattern. Rows, schemas, order and memory estimates are unchanged.
OrderedCursor::next_ordered_batch(ordered_cursor.rs) now builds the chunk in key order withinterleave_record_batchover the(batch, row)pairs it already holds. Before, it ranconcat_batchesand thentake_record_batch, which copied the chunk twice. The key-order check (row_id_ref, also inengine/push/hydrate.rs) now compares borrowed&strvalues and no longer allocates twoStrings per row.fetch_window(engine/operators/hydrate.rs) records each fetched row'sWorkMemory::row_bytesonce, when it holds the row.interleave_bytes/interleave_once(engine/operators/memory.rs) add those sizes per pick. Before, every estimate calledto_data()on each one-row batch for each column and sliced every column of every pick, and the estimate runs at least once per window and again on every halving.KeyGroups(engine/operators/key_groups.rs) numbers distinct Utf8 keys by first appearance. It hashes the whole array with DataFusion'screate_hashesinto ahashbrown::HashTableof(hash, group), the same pattern ashash_join.rs's build side and with the sameHASH_SEED. The key-set block (anti_join.rs::key_set_mask) and the bound-edge source grouping (graph.rs,rows_by_src) use it in place of a stdHashMap<&str, _>(SipHash), one row at a time. The hash join's build side (hash_join.rs) now uses it too, in place of its ownindex_rows/find_row/hash_ids, soKeyGroupsis the one string-keyed table of the operators and ownsHASH_SEED.Closes #1340
Instruments (checked in)
Every cost claim has an
#[ignore = "instrument: …"]test that times the old code against the new on the same input and asserts equal results. Run them in a release build:cargo test --release --locked -p omnigraph-engine --lib -- --ignored --nocapture ordered_rows_against_concat_then_take_cost interleave_estimate_from_row_bytes_cost key_groups_against_a_string_map_costRelease build, pinned toolchain 1.97.1, best of 30 (chunk, estimate) or 10 (key groups), on an Apple-silicon Mac:
ordered_rows_against_concat_then_take_cost: 8 × 1,024 wide rows, shuffledinterleave_estimate_from_row_bytes_cost: 1,024 one-row batches, 2,048 pickskey_groups_against_a_string_map_cost: 100k outer keys (50k distinct), 1M probesA window is estimated at least twice (
copy_bytes, theninterleave_once) and again on every halving, so the one sizing pass is already cheaper than the two estimates it replaces. These are micro-benchmarks; no end-to-end export, hydration or key-set query was timed.Reproduction
The issue reports costs, not a wrong result, so no test can fail on main: every output is identical. I reproduced the costs with a scratch benchmark (release build, on the pinned crates arrow 58.4.1, datafusion-common 54.0.0 and hashbrown 0.17), on the same gVisor box as main
b125d97a1ab3be9ee5c97243dc0f0cf1a3525baf:assert_eq!to main'sEach figure is the best of 30 runs (1) or 10 runs (3), from two separate executions. I did not re-measure (2) on this box: its estimate code is private to the engine. The issue measured 1,598 µs → 37 µs for the estimate over 1,024 one-row batches with 2,048 picks. The equivalence test below pins that the result is unchanged.
Verdict: partly reproduced. The costs for (1) and (3) reproduce. (2) is shown by code inspection and the equality test, not timed. There is no behavioural defect to reproduce.
Root cause
next_ordered_batchcopied the hydrated batches into one batch and then took from it. The peak was the chunk, its concatenation and the ordered copy: three chunk copies where two are enough.interleave_estimatewalked columns × picks, callingto_data()andslice_memory_sizefor each.copy_bytescalls it for every halving of a window andinterleave_oncecalls it again, so the same rows were sized repeatedly.key_set_maskand the bound-edgerows_by_srchashed each key one row at a time with std's SipHashHashMap.Decisions and alternatives
KeyGroupsfor the three string-keyed tables (hash-join build, key-set block, bound-edge sources), not three hand-written ones. It stores no string copies: a group keeps the row where its key first appears, and two keys with one hash stay separate groups, compared by their strings.group_startsintogroup_rows) instead of aVec<u32>per key. Each group's rows keep their input order, so output order within an edge row is unchanged. The scan's key list now comes in first-appearance order instead ofHashMapiteration order. The scan treats it as a set.GroupValueswithGroupsAccumulator(the issue's follow-up). It would move the key-set fold towardAggregateExec, but it depends on query: correlated block aggregates fold row by row; decide their float semantics, then use DataFusion's grouped accumulators #1260's float decision. In the issue's benchmark it was also slower than theHashTablepath (27.9 against 23.6 ns/row).HashMap<&str, _>. That would still hash one row at a time and would add a third hashing pattern besidehash_join's.interleave_bytes/interleave_oncetakerow_bytes: &[Vec<usize>](per batch, per row) from the caller, who computes them once while it holds the rows. The per-pick+16, the per-column+128, the scratch estimate and admission are unchanged.Memory charges
key_set_mask: wasentries::<(&str, u32)>+entries::<u32>per outer row (120 B). NowKeyGroups::charge(104 B):entries::<(u64, u32)>for the table,entries::<u32>for first rows, plus the exact hash buffer and per-row groups.entries::<(&str, Vec<u32>, u32, String)>(292 B/row). NowKeyGroups::charge+entries::<(u32, u32, String)>(236 B/row), plus the unchanged string charge.entries::<(u64, u32)>per row (68 B) kept, plus the hash buffer released after the build. NowKeyGroups::group_bytes(88 B: the table entry and a first row per group) kept, plus the same released hash buffer. The probe side is unchanged.u64hash and theOption<u32>group, inanti_join.rshashes/inner_groupsandgraph.rsprobe_hashes/probe_groups), beside the existing 4 Btags. Each buffer is reused and bounded by one input batch.Tests
engine::operators::key_groups::tests::key_groups_number_keys_like_a_string_map_issue_1340(new): groups equal a string map's (first-appearance order, nulls excluded, empty and prefix-sharing keys), probes find exactly the built keys, and a sliced probe hashes its own rows.engine::operators::memory::row_bytes_tests::interleave_estimate_from_row_bytes_equals_the_per_slice_sum_issue_1340(new): the estimate from row bytes equals the per-column, per-pick slice sum plus scratch. It covers one-row, multi-row and sliced batches, a repeated pick and an empty pick list, and refuses a pick of an absent batch or row.engine::operators::key_groups::tests::key_groups_keep_the_first_row_and_resolve_a_collision(moved fromhash_join.rs'sbuild_table_keeps_the_first_row_and_resolves_a_collision): forced hash collisions stay separate groups, the first row of a duplicate is kept, and a null never joins a group.ordered_cursor::ordered_rows_tests::ordered_rows_equal_concat_then_take(new): the interleaved chunk equals the old concat-then-take, with a sliced batch and a repeated pick.engine::operators::expand_stream::tests::bound_pairs_refuse_a_null_source_id(new): a null bound source id is refused and nothing is emitted.take_estimatetest inmemory.rs) move to the new signature unchanged.integration/export.rs(13 passed, including the wide-row ordered export),integration/changes.rs(60) andintegration/merge_fast_forward.rs(20) for the ordered cursor;integration/engine_v2_memory.rs(31) for hydration chunk bounds and halving; the GQT corpus (467 cases,expect same as v1steps compare rows with the reference engine) for key-set blocks and bound edges.changelog.d/export-hydration-keyset-cpu.performance.md.Local gates
Review round (
be615840), toolchain 1.97.1 as pinned:cargo fmt --all --checkandcargo dev-clippy: clean.cargo dev-test --lib --test integrationfiltered to the owners of these paths (key groups, hash join, anti-join, report, row bytes, ordered rows, bound pairs, and integrationexport::,engine_v2_memory::,changes::,merge_fast_forward::,traversal_indexed::,engine_v2::,aggregation::,engine_v2_plan_replay::): 247 passed, the 3 instruments ignored. The instruments pass in debug and release.First round (base
b125d97a):cargo fmt --all --check: clean.cargo dev-clippy: clean.cargo clippy -p omnigraph-gqt --all-targets --locked -- -D warnings -W clippy::dbg_macro: clean.cargo test -p omnigraph-gqt --locked: 467 corpus cases passed, harness tests passed.cargo test -p omnigraph-engine --lib --features failpoints: 639 passed, 1 failed:db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun. It fails only because the sandbox runs as root, so achmod 000object is still readable. The failure is unrelated to this change, and CI is the authoritative run.export::,engine_v2_memory::,changes::,merge_fast_forward::,engine_v2::,engine_v2_plan_replay::,traversal_indexed::,aggregation::: all passed.cargo test --lockedfromcrates/omnigraph-dst: passed (scenarios 56 passed, 30 ignored fleet/hunt instruments), includingdst_bench_cost_count_golden, so no storage-cost golden line changed: this change reads and writes nothing new.Risks and what was not verified
"". A bound source is a node, whose id is never null, so it now refuses a null withmanifest_internalinstead of dropping the row (invariant 8). The key-set block keeps treating a null key as "no matches", its documented contract.Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261110T1912Z-12d37b