Repository navigation
Conversation
This was referenced Sep 19, 2026
Six lance_surface_guards probes and one CLI test from PR #606, applied on main; none of them depends on the RFCs being accepted: - fts_statistics_scope_can_reverse_ranking: eligibility and the BM25 scoring corpus are independent (an external row mask keeps the scores an index rebuild over the subset would reverse). - nfc_preprocessing_requires_an_explicit_bounded_integration: the public tokenizer can be reused after NFC, but no index-builder option adds it. - native_fuzzy_bm25_rewards_a_rare_expansion and native_bm25_idf_can_round_a_common_term_to_zero: why native BM25 is not the proposed scorer (rare expansion outweighs the exact term; f32 IDF of a common term rounds to zero at N = 2^24). - lance_provider_scan_payloads_need_accounting_beyond_the_session_pool and native_scheduler_drop_distinguishes_queued_and_dispatched_reads: the session pool does not bound decoded Lance batches, and dropping a result future cancels a queued read but not a dispatched one. - cli_data: a recorded model label does not freeze a provider's output at a snapshot; explicit vectors make no provider call and a mismatched label is refused. The IDF reference values use f64::ln_1p instead of the pinned libm dev-dependency the PR carried; these probes need a positive reference, not bit identity.
ragnorc
force-pushed
the
search-substrate-guards
branch
from
September 25, 2026 08:33
520810b to
1e0bed4
Compare
Contributor
Author
|
Closing this. The Lance guards here pin substrate facts that only the search RFCs rely on, and those RFCs were rewritten from scratch in #791, #792 and #793. Each guard will return in the implementation PR that first depends on it, so it lands next to the code it protects instead of ahead of it. The code stays reachable at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Pins seven substrate facts behind the search RFCs (PR #606) as ordinary tests on main, so the RFCs can cite them and none of them waits on RFC acceptance. Six
lance_surface_guardsprobes and one CLI test, taken from #606 atce5a3012and applied onto main.fts_statistics_scope_can_reverse_ranking: eligibility and the BM25 scoring corpus are independent; an external row mask keeps the scores an index rebuild over the subset would reverse.nfc_preprocessing_requires_an_explicit_bounded_integration: the public tokenizer can be reused after NFC, but no index-builder option adds it.native_fuzzy_bm25_rewards_a_rare_expansionandnative_bm25_idf_can_round_a_common_term_to_zero: why native BM25 is not the proposed scorer (a rare expansion outweighs the exact term; the f32 IDF of a common term rounds to zero at N = 2^24).lance_provider_scan_payloads_need_accounting_beyond_the_session_poolandnative_scheduler_drop_distinguishes_queued_and_dispatched_reads: the session pool does not bound decoded Lance batches, and dropping a result future cancels a queued read but not a dispatched one.cli_data::recorded_model_label_does_not_freeze_provider_output_at_a_snapshot: a recorded model label is not an immutable encoder; explicit vectors make no provider call; a mismatched label is refused before any call.The IDF reference values use
f64::ln_1pinstead of the pinnedlibmdev-dependency the RFC branch carried; these probes need a positive reference, not bit identity.Backing issue / RFC
Checklist
Local verification
cargo test -p omnigraph-engine --locked --test lance_surface_guards -- lance_provider_scan_payloads native_scheduler_drop fts_statistics_scope nfc_preprocessing native_fuzzy_bm25 native_bm25_idf— 6 passed on main8281807bcargo test -p omnigraph-cli --locked --test cli_data recorded_model_label— 1 passedcargo fmton the touched file — cleancargo clippy --workspace --all-targets— not run locally; the probes are the rfc: search plan truth, retrieval algebra, analyzed lexical search, and the query kernel #606 code plus one closing brace and threef64::ln_1pcallsNotes for reviewers
The two resource probes (session pool, scheduler drop) characterize behavior that only the engine version 2 memory/admission component (RFC 0067) will depend on; if you would rather they wait for that RFC, drop those two and the rest stands alone.