Skip to content

A declared full-text index goes missing after a type's first load or a property add/rename #841

Open

Description

@ragnorc

Behaviour

RFC 0047 step 3 (#747) refuses a full-text call (search, fuzzy, match_text, bm25) on a declared full-text index that has no built segment, with FullTextIndexRequired (HTTP 409), because Lance's flat match without a segment uses a bare tokenizer that neither folds case nor stems. Before step 3 those calls answered from that tokenizer, silently differently from an indexed search.

Two ordinary operations leave a declared full-text index with no segment, so with step 3 they take full-text search on a type offline until someone runs an index build:

  1. A type's first rows. init and schema apply create tables empty, ensure_indices/optimize skip empty tables, and load/mutate build no index (invariant 7, no index builds on the write path). The first load into a new type therefore refuses full-text calls until a build.
  2. Schema apply adding or renaming a property. SchemaMigrationStep::AddProperty and RenameProperty put the type in rewritten_tables, which schema_apply.rs rewrites with stage_overwrite. A Lance Overwrite drops every index of the table (Lance's own dataset_index.rs test: after an overwrite, load_indices() is empty). The code comments there say reads stay correct "via a scan", which no longer holds for full-text calls.

Reproduced on the step 3 branch: node Doc { slug: String @key title: String @index }, two rows loaded, indexes built, search($d.title, "Deep") returns 2 rows. After apply_schema adds note: String? to Doc, load_indices() on node:Doc is empty and the same query returns FullTextIndexRequired { index: "Doc.title" }. Scalar and vector indexes are dropped the same way; those only cost speed.

Proposals

  1. Build declared full-text indexes when a table is created. Lance 11 builds a full-text index on an empty table (maintenance.rs::full_text_rebuild_reports_empty_builds_but_not_no_work), with an empty fragment bitmap. Rows inserted later fall outside its segments and are matched with its analyzer: on the step 3 branch, after an empty build and an insert of "Deep Learning" and "deep dive", "Deep" finds both rows, "learn" finds "Deep Learning" and "dives" finds "deep dive", and the analyzer certificate check passes. Coverage is then partial, which answers correctly at scan cost until the next optimize. The build would run inside init and inside the schema apply that adds a type, in the same publication. Cost: one empty index build per declared full-text property of each new table. It amends one rule in docs/dev/execution.md: "Schema apply, mutation, and Load publish logical data and index intent only."
  2. Evolve columns instead of overwriting. Lance keeps indexes through Operation::Merge (adding columns; stale entries for rewritten fragments are pruned, the rest retained) and Operation::Project (renaming; TableStore::renamed_schema already uses it for the system-column respelling, which must keep index attachments). Using them for AddProperty and RenameProperty would keep every index of the type and avoid rewriting its data.

To decide

  • Whether step 3 waits for one or both of these, or ships with the refusal documented (run index build after a load into a new type and after a schema apply that adds or renames a property).
  • Whether proposal 1 needs an RFC amendment for the index-intent rule.

Activity

  1. added
    bugSomething isn't working
    acceptedTriaged and validated; open for a PR
    P-highHigh priority
    and removed
    enhancementNew feature or request
    needs-triageAwaiting maintainer triage
    on Oct 4, 2026
  2. ragnorc commented on Oct 16, 2026

    @ragnorc
    ContributorAuthor

    Triage: reproduction and fix assessment

    Summary

    This issue says two ordinary writes leave a declared full-text index with no Lance segment:

    1. the first rows of a type;
    2. a schema apply that adds or renames a property.

    With no segment, Lance matches with a bare tokenizer. That tokenizer does not lowercase and does not stem. On the RFC 0047 step 3 branch, this state makes full-text calls refuse with FullTextIndexRequired (HTTP 409) until an index build.

    Result: not reproduced on current main. Two merged PRs remove both causes:

    Step 3 (#747) has not landed. FullTextIndexRequired is only in the RFC 0047 text, not in code, so the reported 409 cannot occur on main. Instead, I tested the same observable contract on main: a full-text answer must use the index's analyzer. It does.

    One risk is still open for step 3: what "built" means. See "Recommended fix".

    Reproduction

    Main SHA: 9c2b3679e8f66062252e7ed6270a9478b4b77413 (latest at selection). Before posting, main moved to 450f7745 (#1010, change-feed page tokens only). That commit does not touch any file below.

    Environment:

    • clean detached git worktree;
    • Linux x86_64, kernel 6.12;
    • Rust 1.97.1 and Lance 11.0.0 (both pinned by the repository);
    • protoc 3.21.12;
    • local filesystem storage;
    • engine v2, debug test profile.

    The reporter's steps, as a temporary Rust test. I did not push it, and I restored the file after the run. The full text is at the end of this comment.

    • Schema: node Doc { slug: String @key title: String @index }.
    • Rows: d1 "Deep Learning" and d2 "deep dive". The report does not give the row text, so I used the texts from its proposal section.
    • Query: search($d.title, "Deep"). With the analyzer, "Deep" matches both rows. With the bare tokenizer, it matches only d1.
    • Sequence:
      1. Load the rows, then search before any index build.
      2. Run ensure_indices, then search.
      3. Run apply_schema to add note: String?. Read load_indices(), then search.
      4. Run apply_schema to rename title to heading (@rename_from). Read load_indices(), then search.
      5. Run apply_schema to add the type Fresh { slug: String @key body: String @index }. Load its first two rows with no index build. Read load_indices(), then search.

    Why Rust and not GQT: the .gqt format has no schema-apply step. The first-rows half is already a GQT case (see "Existing tests").

    cargo test -p omnigraph-engine --test schema_apply --locked -- --exact \
      tmp_repro_issue_841_reporter_sequence full_text_analyzer_survives_schema_evolution_issue_904 --nocapture
    
    Step Indexes, main 9c2b3679 search(…, "Deep"), main Indexes, pre-fix dc753879 search(…, "Deep"), pre-fix
    Doc first load, no build not printed d1, d2 not printed d1
    after ensure_indices __id_idx, slug_idx, title_idx, 1 fragment each d1, d2 same d1, d2
    add note: String? same 3, coverage kept d1, d2 none d1
    rename title → heading same 3, coverage kept d1, d2 none d1
    added type Fresh, first load body_idx, slug_idx, untrained (0 fragments) d1, d2 none d1

    Output on main:

    841 after ensure_indices: ["__id_idx[__id] fragments=Some(1)", "slug_idx[slug] fragments=Some(1)", "title_idx[title] fragments=Some(1)"]
    841 after add note: ["__id_idx[__id] fragments=Some(1)", "slug_idx[slug] fragments=Some(1)", "title_idx[title] fragments=Some(1)"]
    841 after rename title->heading: ["__id_idx[__id] fragments=Some(1)", "slug_idx[slug] fragments=Some(1)", "title_idx[heading] fragments=Some(1)"]
    841 added type Fresh, empty: []
    841 Fresh after first load: ["body_idx[body] fragments=Some(0)", "slug_idx[slug] fragments=Some(0)"]
    test full_text_analyzer_survives_schema_evolution_issue_904 ... ok
    test tmp_repro_issue_841_reporter_sequence ... ok
    test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 31 filtered out
    

    Proof before the fix. I ran the same test at dc753879, the parent of #871, so before both #871 and #909, on the same Lance 11.0.0. I had to add one missing use lance::index::DatasetIndexExt; line for that older file to compile.

    assertion `left == right` failed: before any index build
      left: ["d1"]
     right: ["d1", "d2"]
    

    I changed the first assertion to a print. The add step then failed:

    841 after add note: []
    ... panicked at ...: add property dropped every index
    

    I changed all assertions to prints. The full sequence then printed:

    841 before any index build: ["d1"]
    841 after ensure_indices: ["__id_idx[__id] fragments=Some(1)", "slug_idx[slug] fragments=Some(1)", "title_idx[title] fragments=Some(1)"]
    841 search after ensure_indices: ["d1", "d2"]
    841 after add note: []
    841 search after add property: ["d1"]
    841 after rename title->heading: []
    841 search after rename property: ["d1"]
    841 added type Fresh, empty: []
    841 Fresh after first load: []
    841 search first load of an added type: ["d1"]
    

    Existing tests on main also pass:

    cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- --exact case::text_search_fuzzy_and_match_text_filters.gqt
    # ok: the `Fresh` steps cover the first rows, an update and a delete
    cargo test -p omnigraph-engine --test search --locked -- --exact full_text_analyzer_holds_after_first_rows_and_overwrites_issue_904
    # ok: first rows, an overwrite load, and a type's first rows on a branch
    

    Differences from the report:

    • Symptom. The report was on the step 3 branch, where the symptom is the 409 FullTextIndexRequired. Main has no step 3. Before the fix, the main symptom was a silent case-sensitive answer (["d1"]), the Unindexed full-text search is case-sensitive #747/Schema apply that rewrites a table drops all its indexes; full-text search then misses matches #904 class. The mechanism is the same; only the surface is different.
    • Indexes after the add. The report says load_indices() on node:Doc is empty after the add. On main, all three indexes stay, with their coverage. Before the fix, the list was empty, as reported.
    • Creation paths. Two facts in the report still hold on main. The table of an added type is still created empty with no segment (schema_apply.rs L747–L761), and ensure_indices still skips empty tables. But the first write now adds the segment, so no version with rows is without one.

    Root cause, and how main fixes it

    1. No segment means a bare tokenizer. In Lance 11, the flat match takes its tokenizer from the first full-text segment. With no segment, it uses SimpleTokenizer (fts.rs L2541–L2557, L1388–L1393). Lance main still does this today.
    2. Overwrite drops every index (manifest_build.rs L788). Before perf(schema): apply property changes as metadata-only commits; optimize erases dropped columns #871, schema apply rewrote a changed table with stage_overwrite. Now stage_schema_evolution commits a Project (rename, drop) and then a Merge (add). Lance's retain_relevant_indices (index_maintenance.rs L221) keeps each index whose field survives, and it matches fields by field id, so a rename keeps its index (schema_apply.rs L821–L842).
    3. First rows. ensure_indices skips empty tables, and writes build no postings (invariant 7). fix(engine): every written version keeps a declared full-text index's analyzer #909 makes TableStore::commit_staged_detached take the declared full-text columns as a required argument:
      • full_text_columns_without_segment reads the base version's index metadata and the operation. From them it finds the columns that the new version will hold no segment for.
      • For those columns, the gateway chains a CreateIndex with an untrained segment: train(false), an empty fragment bitmap, and the analyzer certificate. The chained commit uses the same witness.
      • Code: table_store.rs L840–L898, L3976–L4001, L4028–L4062.
      • Every production writer passes the declared columns: load and mutate staging, merge, optimize and schema apply. The system-column upgrade passes none, because its rename-only Project keeps every index.
      • The Lance guard fts_untrained_segment_applies_its_analyzer_to_rows_it_does_not_cover (test(lance): guard an untrained full-text segment and stale delta-segment entries #905) pins one fact: an untrained segment gives the same rows and BM25 scores as a full build.

    Recommended fix (remaining work)

    Do not add more engine code for this issue. Main fixes the defect by design, at the one write door. The remaining work is a definition in step 3.

    1. Step 3 must refuse on "no segment", not on "no postings".

    RFC 0047 says FullTextIndexRequired fires when "the property is declared but its index has no built segments" (L154–L158). The codebase now uses "built" to mean "has postings". built_index_names and has_fts_postings_on leave out untrained segments (dataset_index.rs L243–L272, omnigraph-catalog/src/lib.rs L532–L558).

    If step 3 reads its planning fact from postings, the refusal comes back until optimize. It would fire after every first load into a type, every overwrite load and every newly declared @index. That is this issue again. The refusal exists because the bare tokenizer gives wrong answers, and an untrained segment removes that cause. So:

    • Read the step 3 fact from segment presence (has_fts_index_on, which counts untrained segments). Keep the RFC 0043 certificate check on every segment that a query selects.
    • Amend RFC 0047 to say "has no full-text segment at the query's snapshot (no postings and no analyzer declaration)".
    • Keep text_search_fuzzy_and_match_text_filters.gqt green with step 3 on. Its Fresh steps fail if step 3 checks for postings. If the format can make a snapshot with no segment, add a step 3 GQT case that refuses only that snapshot.

    After this change, FullTextIndexRequired fires only for table versions written before #909: tables in existing graphs that no write has touched since, and old snapshots. The next write to such a table adds the segment, and omnigraph optimize builds the postings. Old snapshots stay as they are. The step 3 PR should document this.

    2. Small documentation follow-up. docs/dev/execution.md L1025 still says "Schema apply, mutation, and Load publish logical data and index intent only." Writers now also publish an analyzer declaration, without postings. docs/dev/writes.md and docs/dev/lance.md already describe this. One clarifying sentence would close the gap that this issue names.

    I do not think #909's shape needs an RFC amendment. It builds no postings on the write path, so the deny-list item "synchronous vector or FTS rebuilds on a content-write path" does not apply. Maintainers decide this.

    3. Optional test hardening. Main splits the coverage between two tests:

    No test pins the analyzer answer after ensure_indices followed by an unrelated add, which is the reporter's exact sequence. To close this gap, extend the #904 test: after ensure_indices, add a nullable property, then check search and segments again. This is cheap and uses the same test.

    Alternatives and tradeoffs

    Option Status Tradeoff
    A. Build an empty full-text index at table creation (proposal 1 as written) Not chosen; #909 does it at the write gateway Covers only the creation paths. Overwrite loads, a branch's first rows and future writers each need their own call site. Higher liability.
    B. Declare the analyzer at the commit gateway (#909) On main One door with a required argument, so every writer is covered. Cost: one more small detached commit and one certificate file for each first write, overwrite load and full-text column add. Lance also reads the index section earlier (measured in #909). Depends on one Lance behaviour, which a guard pins.
    C. Project/Merge instead of Overwrite (proposal 2, #871) On main Keeps every index and writes no data file. Values of a dropped property stay on disk until optimize (documented).
    D. Apply the analyzer at query time to rows with no index (RFC PR #792, or upstream lance#5889) Open; no upstream PR Most correct by design: correctness does not depend on any physical segment (invariant 7). Needs BM25 statistics parity and a scorer. Would remove the step 3 refusal. This is the long-term direction.

    Liability. B and C have low ongoing liability, because the required argument makes every new writer state its declared columns. The main risk is a Lance change to one of two behaviours: the flat path taking its tokenizer from the first segment, or the handling of an empty fragment bitmap. The #905 guards fail at that Lance bump.

    Lance upstream assessment

    Related work

    Supersession

    #904 has the same mechanism, and #909 fixed it. This issue also holds the step 3 framing and its "To decide" items, and those are still open decisions for #747. So I do not close this issue as a duplicate.

    Suggestion: close this issue as completed by #871 and #909 when the step 3 owner accepts recommendation 1. Alternatively, keep it open as a checklist item for step 3.

    Limitations

    Temporary test used (added to crates/omnigraph/tests/schema_apply.rs, not pushed)
    #[tokio::test]
    #[cfg_attr(feature = "failpoints", serial_test::parallel)]
    async fn tmp_repro_issue_841_reporter_sequence() {
        let dir = tempfile::tempdir().unwrap();
        let uri = dir.path().to_str().unwrap();
        let v1 = "node Doc {\n    slug: String @key\n    title: String @index\n}\n";
        let db = helpers::session(Omnigraph::init(uri, v1).await.unwrap());
        db.load_jsonl(
            r#"{"type":"Doc","data":{"slug":"d1","title":"Deep Learning"}}
    {"type":"Doc","data":{"slug":"d2","title":"deep dive"}}"#,
            LoadMode::Merge,
        )
        .await
        .unwrap();
        let search = |ty: &'static str, column: &'static str| {
            let db = &db;
            async move {
                let source = format!(
                    "query docs($q: String) {{ match {{ $d: {ty} search($d.{column}, $q) }} return {{ $d.slug }} }}"
                );
                first_column_sorted(
                    &query_main(db, &source, "docs", &params(&[("q", "Deep")]))
                        .await
                        .unwrap(),
                )
            }
        };
        let indexes = |table: &'static str| {
            let db = &db;
            async move {
                let ds = open_pinned_dataset_for_test(db, "main", table).await;
                let schema = ds.schema().clone();
                let mut out = ds
                    .load_indices()
                    .await
                    .unwrap()
                    .iter()
                    .map(|index| {
                        let cols: Vec<String> = index
                            .fields
                            .iter()
                            .filter_map(|id| schema.field_by_id(*id).map(|f| f.name.clone()))
                            .collect();
                        format!(
                            "{}[{}] fragments={:?}",
                            index.name,
                            cols.join(","),
                            index.fragment_bitmap.as_ref().map(|b| b.len())
                        )
                    })
                    .collect::<Vec<_>>();
                out.sort();
                out
            }
        };
    
        assert_eq!(search("Doc", "title").await, ["d1", "d2"], "before any index build");
        db.ensure_indices().await.unwrap();
        eprintln!("841 after ensure_indices: {:?}", indexes("node:Doc").await);
        assert_eq!(search("Doc", "title").await, ["d1", "d2"]);
    
        let v2 = "node Doc {\n    slug: String @key\n    title: String @index\n    note: String?\n}\n";
        assert!(db.apply_schema(v2).await.unwrap().applied);
        let after_add = indexes("node:Doc").await;
        eprintln!("841 after add note: {after_add:?}");
        assert!(!after_add.is_empty(), "add property dropped every index");
        assert!(
            after_add.iter().any(|i| i.contains("[title] fragments=Some(1)")),
            "title postings kept: {after_add:?}"
        );
        assert_eq!(search("Doc", "title").await, ["d1", "d2"], "after add property");
    
        let v3 = "node Doc {\n    slug: String @key\n    heading: String @rename_from(\"title\") @index\n    note: String?\n}\n";
        assert!(db.apply_schema(v3).await.unwrap().applied);
        let after_rename = indexes("node:Doc").await;
        eprintln!("841 after rename title->heading: {after_rename:?}");
        assert!(
            after_rename.iter().any(|i| i.contains("[heading] fragments=Some(1)")),
            "heading postings kept: {after_rename:?}"
        );
        assert_eq!(search("Doc", "heading").await, ["d1", "d2"], "after rename property");
    
        let v4 = "node Doc {\n    slug: String @key\n    heading: String @rename_from(\"title\") @index\n    note: String?\n}\nnode Fresh {\n    slug: String @key\n    body: String @index\n}\n";
        assert!(db.apply_schema(v4).await.unwrap().applied);
        eprintln!("841 added type Fresh, empty: {:?}", indexes("node:Fresh").await);
        db.load_jsonl(
            r#"{"type":"Fresh","data":{"slug":"d1","body":"Deep Learning"}}
    {"type":"Fresh","data":{"slug":"d2","body":"deep dive"}}"#,
            LoadMode::Merge,
        )
        .await
        .unwrap();
        let fresh = indexes("node:Fresh").await;
        eprintln!("841 Fresh after first load: {fresh:?}");
        assert!(
            fresh.iter().any(|i| i.contains("[body] fragments=Some(0)")),
            "first load declares the analyzer: {fresh:?}"
        );
        assert_eq!(search("Fresh", "body").await, ["d1", "d2"], "first load of an added type");
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P-highHigh priorityacceptedTriaged and validated; open for a PRbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions