Repository navigation
Description
Behaviour
RFC 0047 step 3 (#747) refuses a full-text call (search, fuzzy, match_text, bm25) on a declared full-text index that has no built segment, with FullTextIndexRequired (HTTP 409), because Lance's flat match without a segment uses a bare tokenizer that neither folds case nor stems. Before step 3 those calls answered from that tokenizer, silently differently from an indexed search.
Two ordinary operations leave a declared full-text index with no segment, so with step 3 they take full-text search on a type offline until someone runs an index build:
- A type's first rows.
initand schema apply create tables empty,ensure_indices/optimizeskip empty tables, andload/mutatebuild no index (invariant 7, no index builds on the write path). The first load into a new type therefore refuses full-text calls until a build. - Schema apply adding or renaming a property.
SchemaMigrationStep::AddPropertyandRenamePropertyput the type inrewritten_tables, whichschema_apply.rsrewrites withstage_overwrite. A LanceOverwritedrops every index of the table (Lance's owndataset_index.rstest: after an overwrite,load_indices()is empty). The code comments there say reads stay correct "via a scan", which no longer holds for full-text calls.
Reproduced on the step 3 branch: node Doc { slug: String @key title: String @index }, two rows loaded, indexes built, search($d.title, "Deep") returns 2 rows. After apply_schema adds note: String? to Doc, load_indices() on node:Doc is empty and the same query returns FullTextIndexRequired { index: "Doc.title" }. Scalar and vector indexes are dropped the same way; those only cost speed.
Proposals
- Build declared full-text indexes when a table is created. Lance 11 builds a full-text index on an empty table (
maintenance.rs::full_text_rebuild_reports_empty_builds_but_not_no_work), with an empty fragment bitmap. Rows inserted later fall outside its segments and are matched with its analyzer: on the step 3 branch, after an empty build and an insert of "Deep Learning" and "deep dive", "Deep" finds both rows, "learn" finds "Deep Learning" and "dives" finds "deep dive", and the analyzer certificate check passes. Coverage is then partial, which answers correctly at scan cost until the nextoptimize. The build would run insideinitand inside the schema apply that adds a type, in the same publication. Cost: one empty index build per declared full-text property of each new table. It amends one rule indocs/dev/execution.md: "Schema apply, mutation, and Load publish logical data and index intent only." - Evolve columns instead of overwriting. Lance keeps indexes through
Operation::Merge(adding columns; stale entries for rewritten fragments are pruned, the rest retained) andOperation::Project(renaming;TableStore::renamed_schemaalready uses it for the system-column respelling, which must keep index attachments). Using them forAddPropertyandRenamePropertywould keep every index of the type and avoid rewriting its data.
To decide
- Whether step 3 waits for one or both of these, or ships with the refusal documented (run
index buildafter a load into a new type and after a schema apply that adds or renames a property). - Whether proposal 1 needs an RFC amendment for the index-intent rule.
Activity
ragnorc commentedon Oct 16, 2026
Triage: reproduction and fix assessment
Summary
This issue says two ordinary writes leave a declared full-text index with no Lance segment:
With no segment, Lance matches with a bare tokenizer. That tokenizer does not lowercase and does not stem. On the RFC 0047 step 3 branch, this state makes full-text calls refuse with
FullTextIndexRequired(HTTP 409) until an index build.Result: not reproduced on current main. Two merged PRs remove both causes:
Operation::Project,Operation::Merge), and these keep every index. This is proposal 2 of this issue.Step 3 (#747) has not landed.
FullTextIndexRequiredis only in the RFC 0047 text, not in code, so the reported 409 cannot occur on main. Instead, I tested the same observable contract on main: a full-text answer must use the index's analyzer. It does.One risk is still open for step 3: what "built" means. See "Recommended fix".
Reproduction
Main SHA:
9c2b3679e8f66062252e7ed6270a9478b4b77413(latest at selection). Before posting, main moved to450f7745(#1010, change-feed page tokens only). That commit does not touch any file below.Environment:
git worktree;testprofile.The reporter's steps, as a temporary Rust test. I did not push it, and I restored the file after the run. The full text is at the end of this comment.
node Doc { slug: String @key title: String @index }.d1 "Deep Learning"andd2 "deep dive". The report does not give the row text, so I used the texts from its proposal section.search($d.title, "Deep"). With the analyzer,"Deep"matches both rows. With the bare tokenizer, it matches onlyd1.ensure_indices, then search.apply_schemato addnote: String?. Readload_indices(), then search.apply_schemato renametitletoheading(@rename_from). Readload_indices(), then search.apply_schemato add the typeFresh { slug: String @key body: String @index }. Load its first two rows with no index build. Readload_indices(), then search.Why Rust and not GQT: the
.gqtformat has no schema-apply step. The first-rows half is already a GQT case (see "Existing tests").9c2b3679search(…, "Deep"), maindc753879search(…, "Deep"), pre-fixDocfirst load, no buildd1, d2d1ensure_indices__id_idx,slug_idx,title_idx, 1 fragment eachd1, d2d1, d2note: String?d1, d2d1title→headingd1, d2d1Fresh, first loadbody_idx,slug_idx, untrained (0 fragments)d1, d2d1Output on main:
Proof before the fix. I ran the same test at
dc753879, the parent of #871, so before both #871 and #909, on the same Lance 11.0.0. I had to add one missinguse lance::index::DatasetIndexExt;line for that older file to compile.I changed the first assertion to a print. The add step then failed:
I changed all assertions to prints. The full sequence then printed:
Existing tests on main also pass:
Differences from the report:
FullTextIndexRequired. Main has no step 3. Before the fix, the main symptom was a silent case-sensitive answer (["d1"]), the Unindexed full-text search is case-sensitive #747/Schema apply that rewrites a table drops all its indexes; full-text search then misses matches #904 class. The mechanism is the same; only the surface is different.load_indices()onnode:Docis empty after the add. On main, all three indexes stay, with their coverage. Before the fix, the list was empty, as reported.schema_apply.rsL747–L761), andensure_indicesstill skips empty tables. But the first write now adds the segment, so no version with rows is without one.Root cause, and how main fixes it
SimpleTokenizer(fts.rsL2541–L2557, L1388–L1393). Lancemainstill does this today.Overwritedrops every index (manifest_build.rsL788). Before perf(schema): apply property changes as metadata-only commits; optimize erases dropped columns #871, schema apply rewrote a changed table withstage_overwrite. Nowstage_schema_evolutioncommits aProject(rename, drop) and then aMerge(add). Lance'sretain_relevant_indices(index_maintenance.rsL221) keeps each index whose field survives, and it matches fields by field id, so a rename keeps its index (schema_apply.rsL821–L842).ensure_indicesskips empty tables, and writes build no postings (invariant 7). fix(engine): every written version keeps a declared full-text index's analyzer #909 makesTableStore::commit_staged_detachedtake the declared full-text columns as a required argument:full_text_columns_without_segmentreads the base version's index metadata and the operation. From them it finds the columns that the new version will hold no segment for.CreateIndexwith an untrained segment:train(false), an empty fragment bitmap, and the analyzer certificate. The chained commit uses the same witness.table_store.rsL840–L898, L3976–L4001, L4028–L4062.Projectkeeps every index.fts_untrained_segment_applies_its_analyzer_to_rows_it_does_not_cover(test(lance): guard an untrained full-text segment and stale delta-segment entries #905) pins one fact: an untrained segment gives the same rows and BM25 scores as a full build.Recommended fix (remaining work)
Do not add more engine code for this issue. Main fixes the defect by design, at the one write door. The remaining work is a definition in step 3.
1. Step 3 must refuse on "no segment", not on "no postings".
RFC 0047 says
FullTextIndexRequiredfires when "the property is declared but its index has no built segments" (L154–L158). The codebase now uses "built" to mean "has postings".built_index_namesandhas_fts_postings_onleave out untrained segments (dataset_index.rsL243–L272,omnigraph-catalog/src/lib.rsL532–L558).If step 3 reads its planning fact from postings, the refusal comes back until
optimize. It would fire after every first load into a type, every overwrite load and every newly declared@index. That is this issue again. The refusal exists because the bare tokenizer gives wrong answers, and an untrained segment removes that cause. So:has_fts_index_on, which counts untrained segments). Keep the RFC 0043 certificate check on every segment that a query selects.text_search_fuzzy_and_match_text_filters.gqtgreen with step 3 on. ItsFreshsteps fail if step 3 checks for postings. If the format can make a snapshot with no segment, add a step 3 GQT case that refuses only that snapshot.After this change,
FullTextIndexRequiredfires only for table versions written before #909: tables in existing graphs that no write has touched since, and old snapshots. The next write to such a table adds the segment, andomnigraph optimizebuilds the postings. Old snapshots stay as they are. The step 3 PR should document this.2. Small documentation follow-up.
docs/dev/execution.mdL1025 still says "Schema apply, mutation, and Load publish logical data and index intent only." Writers now also publish an analyzer declaration, without postings.docs/dev/writes.mdanddocs/dev/lance.mdalready describe this. One clarifying sentence would close the gap that this issue names.I do not think #909's shape needs an RFC amendment. It builds no postings on the write path, so the deny-list item "synchronous vector or FTS rebuilds on a content-write path" does not apply. Maintainers decide this.
3. Optional test hardening. Main splits the coverage between two tests:
apply_schema_column_changes_keep_data_files_ids_and_index_coverage(perf(schema): apply property changes as metadata-only commits; optimize erases dropped columns #871) pins index coverage after an add, a rename and a drop.full_text_analyzer_survives_schema_evolution_issue_904pins the analyzer answer on an index that has no postings.No test pins the analyzer answer after
ensure_indicesfollowed by an unrelated add, which is the reporter's exact sequence. To close this gap, extend the #904 test: afterensure_indices, add a nullable property, then checksearchandsegmentsagain. This is cheap and uses the same test.Alternatives and tradeoffs
Project/Mergeinstead ofOverwrite(proposal 2, #871)optimize(documented).Liability. B and C have low ongoing liability, because the required argument makes every new writer state its declared columns. The main risk is a Lance change to one of two behaviours: the flat path taking its tokenizer from the first segment, or the handling of an empty fragment bitmap. The #905 guards fail at that Lance bump.
Lance upstream assessment
main(68b8d2d0d0f0, 2026-10-16) still usesdefault_text_tokenizerwhen "no on-disk FTS segment exists", andOverwritestill setsfinal_indices = Vec::new(). The 12.0.0 and 13.0.0 release notes change neither one. No upstream release fixes or breaks the fix(engine): every written version keeps a declared full-text index's analyzer #909 design.calculate_included_fragsrefuse for inverted indexes, so Lance does not compute a wrong fragment set. This protects the empty fragment bitmap thatis_untrained_full_textdepends on. Good for this design; watch it.fuzzy()has its own analyzer gap, whether or not a segment exists.Related work
search.rs::full_text_analyzer_holds_after_first_rows_and_overwrites_issue_904).Supersession
#904 has the same mechanism, and #909 fixed it. This issue also holds the step 3 framing and its "To decide" items, and those are still open decisions for #747. So I do not close this issue as a duplicate.
Suggestion: close this issue as completed by #871 and #909 when the step 3 owner accepts recommendation 1. Alternatively, keep it open as a checklist item for step 3.
Limitations
dc753879, before both PRs. I did not run the commit between perf(schema): apply property changes as metadata-only commits; optimize erases dropped columns #871 and fix(engine): every written version keeps a declared full-text index's analyzer #909. Which PR fixed which half comes from code reading and from fix(engine): every written version keeps a declared full-text index's analyzer #909's own before/after evidence.title_idxonheading). Lookups use the field id, so the answers do not change.Temporary test used (added to
crates/omnigraph/tests/schema_apply.rs, not pushed)