Skip to content

test(search): pin the substrate facts behind the search RFCs - #751

Closed
ragnorc wants to merge 1 commit into
mainfrom
search-substrate-guards
Closed

ragnorc wants to merge 1 commit into
mainfrom
search-substrate-guards

Conversation

@ragnorc

@ragnorc ragnorc commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

What & why

Pins seven substrate facts behind the search RFCs (PR #606) as ordinary tests on main, so the RFCs can cite them and none of them waits on RFC acceptance. Six lance_surface_guards probes and one CLI test, taken from #606 at ce5a3012 and applied onto main.

  • fts_statistics_scope_can_reverse_ranking: eligibility and the BM25 scoring corpus are independent; an external row mask keeps the scores an index rebuild over the subset would reverse.
  • nfc_preprocessing_requires_an_explicit_bounded_integration: the public tokenizer can be reused after NFC, but no index-builder option adds it.
  • native_fuzzy_bm25_rewards_a_rare_expansion and native_bm25_idf_can_round_a_common_term_to_zero: why native BM25 is not the proposed scorer (a rare expansion outweighs the exact term; the f32 IDF of a common term rounds to zero at N = 2^24).
  • lance_provider_scan_payloads_need_accounting_beyond_the_session_pool and native_scheduler_drop_distinguishes_queued_and_dispatched_reads: the session pool does not bound decoded Lance batches, and dropping a result future cancels a queued read but not a dispatched one.
  • cli_data::recorded_model_label_does_not_freeze_provider_output_at_a_snapshot: a recorded model label is not an immutable encoder; explicit vectors make no provider call; a mismatched label is refused before any call.

The IDF reference values use f64::ln_1p instead of the pinned libm dev-dependency the RFC branch carried; these probes need a positive reference, not bit identity.

Backing issue / RFC

Checklist

  • Change is focused (one logical change: characterization probes)
  • Tests added/updated for behavior changes (the change is tests)
  • Public docs updated if user-facing surface changed (N/A)
  • Reviewed against docs/dev/invariants.md — no Hard Invariant weakened, no deny-list item hit

Local verification

  • cargo test -p omnigraph-engine --locked --test lance_surface_guards -- lance_provider_scan_payloads native_scheduler_drop fts_statistics_scope nfc_preprocessing native_fuzzy_bm25 native_bm25_idf — 6 passed on main 8281807b
  • cargo test -p omnigraph-cli --locked --test cli_data recorded_model_label — 1 passed
  • cargo fmt on the touched file — clean
  • cargo clippy --workspace --all-targets — not run locally; the probes are the rfc: search plan truth, retrieval algebra, analyzed lexical search, and the query kernel #606 code plus one closing brace and three f64::ln_1p calls

Notes for reviewers

The two resource probes (session pool, scheduler drop) characterize behavior that only the engine version 2 memory/admission component (RFC 0067) will depend on; if you would rather they wait for that RFC, drop those two and the rest stands alone.

Six lance_surface_guards probes and one CLI test from PR #606, applied on
main; none of them depends on the RFCs being accepted:

- fts_statistics_scope_can_reverse_ranking: eligibility and the BM25
  scoring corpus are independent (an external row mask keeps the scores an
  index rebuild over the subset would reverse).
- nfc_preprocessing_requires_an_explicit_bounded_integration: the public
  tokenizer can be reused after NFC, but no index-builder option adds it.
- native_fuzzy_bm25_rewards_a_rare_expansion and
  native_bm25_idf_can_round_a_common_term_to_zero: why native BM25 is not
  the proposed scorer (rare expansion outweighs the exact term; f32 IDF of
  a common term rounds to zero at N = 2^24).
- lance_provider_scan_payloads_need_accounting_beyond_the_session_pool and
  native_scheduler_drop_distinguishes_queued_and_dispatched_reads: the
  session pool does not bound decoded Lance batches, and dropping a result
  future cancels a queued read but not a dispatched one.
- cli_data: a recorded model label does not freeze a provider's output at a
  snapshot; explicit vectors make no provider call and a mismatched label
  is refused.

The IDF reference values use f64::ln_1p instead of the pinned libm
dev-dependency the PR carried; these probes need a positive reference, not
bit identity.
@ragnorc
ragnorc force-pushed the search-substrate-guards branch from 520810b to 1e0bed4 Compare September 25, 2026 08:33
@ragnorc

ragnorc commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Closing this. The Lance guards here pin substrate facts that only the search RFCs rely on, and those RFCs were rewritten from scratch in #791, #792 and #793. Each guard will return in the implementation PR that first depends on it, so it lands next to the code it protects instead of ahead of it.

The code stays reachable at 1e0bed40 (this PR's head), which the analyzed lexical search RFC (#792) cites by permalink.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant