Repository navigation
Keep each phrase signature's vector so an alignment run embeds only texts it has not seen - #1186
Merged
WaylandYang merged 2 commits intoOct 10, 2026
Conversation
…exts it has not seen Signed-off-by: evolv3ai <hello@evolv3.ai>
WaylandYang
approved these changes
Oct 10, 2026
WaylandYang
left a comment
Contributor
There was a problem hiding this comment.
Thank you for cut 2. It is the shape we discussed: vectors kept by base, model and text hash, a run embeds only the texts it has not seen, and the prune takes the full set of hashes so a failed batch does not evict good rows. Taking 0110 instead of the gap is the right call now that dev has moved. Checked here on a fresh database: the store test and a_second_run_over_an_unchanged_base_makes_no_embedding_call pass, and the branch merges cleanly. Merging now. Thanks again for seeing #1097 through.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cut 2 of #1097, on top of cut 1 (#1107). Once properties have vectors,
shortlistre-embeds every wide signature on every run, although a signature's text (phrase · first example · its quote) hardly changes between runs and a run may have only a few dozen signatures to decide. The embedding pass becomes a fixed cost per run that grows with the base.What changes
0110_a_signature_is_embedded_once.sql: tablesignature_vectors (kb_id, model, text_hash, embedding), primary key(kb_id, model, text_hash),ON DELETE CASCADEfromknowledge_bases.embeddingis a dimensionlessvector, aschunks.embeddingis. It's read only by key, so there's no HNSW and novector_indexregistration. It's a cache, not ledger data, so it stays outside the 0070 surface and the export.CURRENT_SCHEMA_VERSION98 → 99 (99 files inmigrations/). The migration was drafted as 0107;devhas since moved to 0109 and version 98, and Explore table alignments and adopt their ontology together #1179 and Govern conflicts in the typed graph and preserve human decisions #1131 both claim 0108, so it takes the next number after 0109 rather than the gap. That renumbering is the only change from the draft besides keepingdev's newmodline next to this one intests/store/main.rs.utopia_store::signature_vectors:get,put(one transaction,ON CONFLICT DO NOTHING), andprune, which deletes this base's rows whose hash no longer occurs and rows embedded with another model.shortlist: hashes (SHA-256) the text of every wide signature, prunes with that full set (not only what this run embeds, so a failed batch doesn't evict good rows), loads the cached vectors, and embeds only the misses, deduplicated, in the same batches of 32. The log line reportsreusedandembedded.shortlist_kind_wordsalready embeds only fresh kind words, and sharing the table would need a source column to keep the two prunes apart. The model key isembed_model. If two endpoints serving the same model name is a concern,embed_base_urlcould join the key.How it was checked
a_second_run_over_an_unchanged_base_makes_no_embedding_call(phrase_alignment_tests.rs): with property vectors present, run 1 embeds once, and run 2 over the unchanged base makes no embedding call and asks the model nothing. After switchingembed_model, run 3 embeds again and only the new model's row is left. Against cut 1'sshortlistit fails withleft: 2, right: 1.a_signature_vector_lasts_while_its_text_does(store):getkeyed by model, collisions don't overwrite,prunedrops gone hashes and other models' rows and leaves other bases alone.cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings: clean.UTOPIA_DATABASE_URLandUTOPIA_TEST_REQUIRE_DB=1against a scratch pgvector:pg16 migrated through 0110: the wholeutopia-storesuite (including the 0070 drift guard and the export guards),utopia-cli(includingschema_version_policy_compares_against_current), andutopia-server -- phrase_alignmentall pass. I didn't run the whole workspace locally; CI covers it.This change was drafted with Claude Code (Anthropic) and reviewed by me before submitting.
Before review
git commit -s)cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warningsandcargo test --workspacepasscrates/utopia-store/was tested withUTOPIA_DATABASE_URLset (those tests skip without it)dev, andCURRENT_SCHEMA_VERSIONincrates/utopia-cli/src/main.rsequals the number of files inmigrations/