L3 world model: structure bloats with paraphrase duplicates — entry-level dedupe in mergeEntries is string-exact only

[hitken]

---

name: World model structure entries bloat with paraphrases — string-exact dedupe in mergeEntries never merges semantic duplicates
about: L3 consolidation appends paraphrased/translated variants of the same fact until the 24-entry cap, evicting older distinct knowledge
title: 'L3 world model: structure bloats with paraphrase duplicates — entry-level dedupe is string-exact only'
labels: bug, l3-world-model
assignees: ''

Summary

In apps/memos-local-plugin (Reflect2Evolve V7), each L3 consolidation pass regenerates world-model structure entries via the abstraction LLM. The entry-level merge in core/memory/l3/merge.ts (mergeEntries) dedupes with an exact string key:

function entryKey(e: { label: string; description: string }): string {
  return `${e.label.toLowerCase().trim()}::${e.description.toLowerCase().trim().slice(0, 64)}`;
}

Any paraphrase, rewording, or cross-language translation of the same fact produces a new key → counted as novel knowledge → appended. The only bound is the capacity cap:

return Array.from(byKey.values()).slice(0, 24);

So on a long-lived install every section saturates at 24 near-duplicate entries, and each new consolidation evicts older, genuinely distinct entries to make room for paraphrases of facts already stored.

Observed impact (production install, ~1 month of use)

6 active world models, all sections pinned at exactly 24/24/24 (432 entries total). Sampling one model ("Node.js project environment model", version 468 after 468 consolidation runs):

• "Global npm packages require PATH configuration"
• "Global installation doesn't guarantee PATH access"
• "npm 全局包路径可见性依赖 PATH"
• …20+ lexical variants of the same single fact, all citing the same two evidence IDs

Consequences:

1. Real knowledge loss — the cap turns paraphrase churn into a rolling eviction of older distinct entries
2. Token waste — worldModel.list() / retrieval serializes the full structure; ~65% of it was duplicates (~30KB per environment query in our case)
3. Retrieval bias — duplicated facts mutually reinforce and dominate tier-3 ranking, crowding out single-shot knowledge
4. Runaway LLM cost — each consolidation re-embeds and re-states the duplicates, and the polluted prompt biases the model toward generating more of the same variants

Root cause

Row-level WM merge (chooseMergeTarget) is fine — cosine-based, works. The problem is exclusively entry-level dedupe inside mergeEntries: string-exact key + capacity cap, with no semantic comparison. There is no config knob exposed for it either (algorithm.l3Abstraction controls row-level clustering only).

Suggested fix

Compare entries semantically before appending, e.g. in priority order:

1. Embedding cosine ≥ ~0.88 between label + description (the plugin already has the embedder; cross-language paraphrases are only catchable this way)
2. Char-3-gram Jaccard ≥ ~0.55 as a lexical fallback when no embedder is available
3. Same non-empty evidenceIds set + Jaccard ≥ ~0.25 — entries citing identical evidence strongly imply the same fact

On merge: keep one entry (fresh phrasing is fine — matches current next-wins semantics), union the evidenceIds.

Note for implementers: track entry vectors in a parallel array by position, not a Map keyed on the input objects — merged stores shallow copies ({...e}), so object-identity lookups silently miss and the semantic branch never fires. (This bit us during testing.)

Workaround we shipped locally

• Patched mergeEntries with the 3-tier semantic dedupe above (embedder plumbed through attachL3Subscriber → runL3 → mergeForUpdate)
• One-off cleanup over world_model.structure_json with the same Jaccard + evidence-ID rules: 432 → 150 entries (−65%) across 6 models, no distinct facts lost on manual review

Happy to turn the local patch into a PR if the approach sounds acceptable.