Metamodel + knowledge bundles#50
Draft
jimador wants to merge 30 commits into
Draft
Conversation
jimador
force-pushed
the
feat/metamodel-and-bundles
branch
10 times, most recently
from
12bd5e6 to
a930a0e
Compare
jimador
force-pushed
the
feat/metamodel-and-bundles
branch
3 times, most recently
from
f06ab0b to
747ed5a
Compare
…uthority Makes projection to a graph observable and traceable, and carries source authority through to the projected edges. - GraphProjectionService and RelationBasedGraphProjector report why a projection was skipped or failed instead of failing silently; Projection carries structured failure reasons - ProjectionRecord lineage records what projected where; ProjectionLineageStaleCascade cascades a stale mark to everything a stale proposition projected - ProjectionPolicySupport carries a proposition's source authority onto the projected edge Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…ctors Adds projectors that turn propositions into output for people rather than graphs. - RationaleProjector / LlmRationaleProjector explain why a conclusion holds - ReportProjector / StructuredReportProjector assemble a structured report - SemanticLink / SemanticLinkDiscoverer surface non-obvious two-hop connections Also fills out the KDoc on AlwaysCreateEntityResolver and EscalatingEntityResolver so the default resolver behavior is documented at the call site. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…estion Wires the projection-lineage SPI to a real repository, adds a reference Neo4j adapter, and gives DICE one front door for getting source material in. - RepositoryBackedReconciler resolves whether to create or reuse a graph artifact by looking it up, instead of always creating — what keeps re-projection from duplicating nodes; comes with a seeded-graph integration proof - Neo4jRagPropositionRepository is a reference store backed by the RAG entity store, declaring only the capability fragments it honestly supports - ingestion SPI: IngestionHandler / TextIngestionHandler turn artifacts into chunks; IngestionLedger dedups by content hash; IngestionResult reports per batch - Testcontainers test-scope deps and a Docker Engine api.version pin for the proof Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Adds the read side over the graph: ask for an entity's neighborhood, the path between two entities, or why a proposition is believed. - GraphQuery with GraphNeighborhood, GraphPath, and PropositionLineage - GraphQueryCapable is the store fragment that answers these, and can filter by the source authority carried on each edge - GraphQueryTools exposes the queries as agent tools Includes the canonical-flow harness that runs the whole extract -> resolve -> project -> query path end to end without an LLM or a database. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…tools Adds a router that picks how to retrieve for a given query, with REST and agent surfaces over it. - RetrievalMode and RetrievalRouter route a query to the right strategy, including a hybrid mode that combines vector and graph - DiscoveryQuery / DiscoveryDtos, DiscoveryController, and DiscoveryTools expose the router over REST and as agent tools, registered through DiceRestConfiguration Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…ests modules Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
… cross-module refs Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
DICE shipped only in-memory ProjectionRecordStore and CollectorRecordStore. Add Drivine-backed implementations that persist projection lineage and the collector audit trail as graph nodes, so they survive a restart and stay queryable. The graph-backed projection store also implements a real markStaleByProposition (the SPI default is a no-op), keeping the lifecycle cascade working against a durable store. Both are wired through DiceStorageAutoConfiguration on the existing embabel.dice.store.type=graph flip, default to in-memory otherwise, and are ConditionalOnMissingBean so an application's own bean wins. Reads and writes use parameterized Cypher (MERGE on the natural key for idempotent upserts); row mapping is extracted so it can be unit-tested without a database. Covered by a Neo4j integration test and row-mapper unit tests. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Add docs/design/graph-projection.md (lineage, named outcomes, the stale cascade, idempotent ingestion/reconciliation, and reaching the graph through a port) and docs/design/retrieval-and-discovery.md (store-agnostic graph queries, query-time authority filtering, one router over many retrieval modes, DTO/context isolation, anchorless serendipitous links, and explainability). Add module AGENTS.md for dice-report, dice-ingestion, and dice-integration-tests, and list them in the root guide. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
… and report paths Wire SLF4J loggers through the retrieval router, graph/lineage stores, report projectors, and ingestion so a consuming application can see the decision and persistence paths: - retrieval: log routing mode/topK/depth, result counts, and per-mode degradation when a capability is unsupported - report: log rationale projection and structured-report counts, and semantic-link discovery sizes - lineage/storage: log stale-cascade and reconciliation outcomes, Drivine collector/projection record writes, and auto-configuration wiring - ingestion: log batch start/finish summaries, dedup hits, and extraction failures Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…bel.dice.spi Update this branch's new code (graph projectors, graph/discovery query types, the ingestion artifact model, and their tests) to import the policy SPIs (TrustScorer, AuthorityResolver/AuthorityTier and friends, ConflictType) from the com.embabel.dice.spi package they now live in. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…uides The package map in dice/AGENTS.md skipped query.graph (GraphQuery, GraphNeighborhood, GraphPath, PropositionLineage) and query.discovery (RetrievalRouter, DiscoveryQuery, RetrievalMode) — public API this branch adds — so an agent navigating by the map could not find them. Add both rows and mention graph/discovery retrieval in the root module description. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The sweep-policy types (MarkReason, PropositionMark, StatusTransitionSweepPolicy) live in com.embabel.dice.spi alongside the other lifecycle policies. The storage row mappers, the discovery DTO-leak gate, and the canonical-flow integration tests still pointed at the old projection.memory package; repoint them so every module compiles against the policy SPI. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Add a design note covering the persistence mechanics no existing note explained — backend selection, defense-in-depth dedup, the two-phase save, materialised effective confidence, schema-as-beans, and the scheduled decay tick — with diagrams. Give dice-storage-autoconfigure its own AGENTS.md (the only module that lacked one), and link graph-projection, retrieval-and-discovery, and durable-storage from the README design-notes index and the root navigation guide. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Verified each reviewer finding against the code before acting (two were false positives and left as-is: a Kotlin self-initializer scoping claim, and the intentional, test-covered RELATED_TO fallback). - LlmGraphProjector: pick the source/target mention by the LLM's span first, falling back to role only when no span matches. The combined `span || role` find let an earlier role-matching mention win over the mention the span actually named, producing wrong-direction edges. - GraphProjectionService: isolate each lineage-record write so a flaky record store can't drop the trail for every remaining result after a mid-batch failure. - GraphQuery.whyExplain: honor the context scope on the global findById path so a context-bound query can't return foreign-context lineage. - GraphQueryCapable: the authority-aware overloads now throw when a backend sets honorsAuthorityFilter but doesn't override them, instead of silently returning unfiltered results. - InMemoryProjectionRecordStore: make the stale check-and-set atomic so concurrent calls don't double-count transitions. - Drivine projection/collector stores: skip and log a corrupt row rather than failing the whole all() query on one bad node. - RetrievalRouter.graphPath: log cross-context paths that are dropped so an empty result is distinguishable from a disconnected graph. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…e SPIs Second adversarial-review pass on the graph + retrieval branch. - Drivine projection/collector stores: every findBy* now pushes its predicate into Cypher instead of loading the whole table and filtering in memory, so a single-key lookup no longer scans the entire lineage. Added Neo4j integration tests asserting each finder returns only its matching subset. - GraphProjectionService: reconcile against the pre-persist graph state (a repository-backed reconciler consulted after the write would always see the node and never record PROJECTED), and reference the produced edge (source-[type]->target) as the lineage targetRef rather than just the source node so findByTargetRef resolves to the specific edge. - MarkReason.Custom: reject the reserved stale/duplicate keys (and blanks) at construction so a Custom can't round-trip back as a built-in reason. - Projection rejection messages quote the policy's actual confidence threshold instead of a hardcoded constant. - DiscoveryQuery exposes a caller-set similarityThreshold (default 0.0, clamped) threaded into the vector and hybrid search requests. - Discovery DTO leak test now rejects any raw com.embabel.dice.proposition type, catching an accidentally-exposed enum, not just the exact FQNs. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Make the architecture legible without reading the code, and close the navigation gaps the review found. - New docs/design/architecture.md: a top-level system overview tying the subsystems together (store + trust, extraction, maintenance, projection, query/retrieval/discovery, report) with system, store-SPI, maintenance, retrieval, expose-layer, and graph-schema diagrams. - Enriched the per-subsystem design notes with sequence, class, state, and flow diagrams so each communicates intent visually (55 diagrams total, all parse-validated). - AGENTS.md navigation: add GraphQueryCapable to the capability fragments, DiscoveryController to web.rest, DiscoveryTools/GraphQueryTools to agent, and the Drivine projection/collector record stores + LineageRowMappers to the storage module guide. - proposition-lifecycle: add the pinning primitive; graph-projection: document the three-way reconciliation decision; fix a mermaid label. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…scoped read Projection-health aggregated lineage across every context, leaking other contexts' projection activity into a context-scoped endpoint/tool. - ProjectionRecord carries the context the proposition belongs to. - ProjectionRecordStore gains findByContext; the REST endpoint and agent tool summarize health from findByContext(contextId), not all(). - The durable Drivine store implements findByContext with scoped Cypher, and the in-memory store filters its backing list directly, so no implementation loads the whole table to answer a scoped read. The all()-based SPI defaults are documented as a trivial-store fallback that durable stores MUST override. - Added a Neo4j integration test asserting findByContext returns only the requested context's records. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Versions the schema and guards it against silent drift. - MetamodelVersion, MetamodelDiff, and MetamodelDiffer (Javers-backed) version the schema and diff one version against another - DriftQuarantinePolicy with MentionTypeDriftQuarantinePolicy quarantines a mention type the current metamodel doesn't know about, instead of letting it drift the schema silently - MetamodelConfiguration ties it together Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Exports a slice of knowledge to a portable bundle and imports it back, locally. - KnowledgeBundle with KnowledgeBundleExporter / KnowledgeBundleImporter, both Jackson-backed - the import result reports what was added versus skipped Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Test-only coverage that pairs with the lifecycle, provenance, and agent surfaces. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Add docs/design/metamodel-governance.md (versioned schema stamps, additive-vs-lossy diffing, and non-destructive drift quarantine) and docs/design/knowledge-bundles.md (carrying the full proposition so a reloaded fact keeps its decay anchor, and treating an incoming bundle as untrusted). Add module AGENTS.md for dice-metamodel and dice-bundle, and list them in the root guide. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…dle paths Wire SLF4J loggers through the bundle export and metamodel diff/wiring seams: - bundle: log export proposition counts and context - metamodel: log diff result counts (added/removed/modified types and relationships) and which default policy implementations get wired Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
JacksonKnowledgeBundleImporter guarded the import size with serialised.length, which counts UTF-16 code units. The constant, the log message, and the failure reason all say "bytes", so a bundle of multi-byte characters could be up to ~2x the intended limit yet still pass. Measure the real UTF-8 byte length instead. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
This branch adds a dedicated DecayStatusPolicyTest in the common test package. Now that DecayStatusPolicy lives in spi, mirror its test there too so it accesses the type same-package, matching the other spi tests. No assertion changes. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The metamodel-governance and knowledge-bundles design notes existed but weren't linked anywhere. Add them to the README design-notes index and the root navigation guide so the governance and transfer subsystems are discoverable. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
jimador
force-pushed
the
feat/metamodel-and-bundles
branch
from
747ed5a to
691debd
Compare
Compare entity-type label and property sets directly in JaversMetamodelDiffer instead of a space-joined string projection. The string form could collapse two genuinely different sets (a label or property whose name contains the delimiter) into a false 'unchanged', silently dropping a lossy modification and skipping quarantine. Removes the now-dead TypeShapeSnapshot machinery. Merge same-named domain types by union when stamping a MetamodelVersion. A DataDictionary can hold two types that share a name but differ in shape, and associate() silently kept only the last — dropping the other's labels/properties from the content hash so their later removal would go undetected. groupBy + union keeps the fingerprint complete. Count an OVERWRITE that replaces a pre-existing proposition separately from a net-new import and leave a note for it, so an idempotency audit can see the destructive write. Dedup by id within a single bundle so a repeated id can't double-write or double-count; the repeat is skipped with a note. Document that SKIP_EXISTING's skip is best-effort under concurrent imports to the same store. Give the drift sweep's already-quarantined pass-through an observable trace: count it apart from genuinely-conforming propositions and surface the number in the summary log. Tests: differ delimiter-collision regression; same-named-type union merge; bundle overwrite counting and within-bundle id dedup; a round-trip test asserting the abstraction, pinning, and reinforcement fields survive; and a status assertion that a re-swept proposition stays STALE. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
jimador
force-pushed
the
feat/metamodel-and-bundles
branch
from
691debd to
11fdd8a
Compare
…cy-safe The bundle importer's SKIP_EXISTING path read the store then wrote, so two concurrent imports of the same id could both see 'absent' and both write, losing the skip. Add a saveIfAbsent primitive to the PropositionStore SPI: it writes only when the id is absent and returns null otherwise. The SPI default stays read-then-write (documented best-effort); the in-memory store overrides it with ConcurrentHashMap.putIfAbsent and the Drivine store runs the check and insert inside its existing stripe-locked transaction, both genuinely atomic. The event-emitting decorator forwards to the delegate's saveIfAbsent so the atomic guarantee survives the decorator and a single PropositionPersisted fires only on a real insert. The importer's SKIP_EXISTING path now writes through saveIfAbsent. Tests: a cross-backend contract test (run against both the in-memory and the Neo4j store) proving insert-once-by-id; an in-memory concurrency test proving exactly one writer wins a contended id; and decorator tests proving saveIfAbsent emits on insert and stays silent on a skip. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The test was disabled because the runtime wiring that fires a trigger-bound @action was considered unverifiable here. It is verifiable and deterministic: build the agent straight from the annotated bean with AgentMetadataReader, drop a PropositionPersisted on the blackboard as the last result, and let the in-process GOAP planner run the single triggered action — no LLM, no threads. Adding @AchievesGoal gives the planner a goal to plan toward (the trigger precondition is excluded from the goal, so it stays achievable once the action runs). Now asserts the action actually fired. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Metamodel + knowledge bundles
Last of the four-PR stack. Stacked on #49 — review after #47–#49 merge; until then the diff
includes their commits.
Carves out two modules:
dice-metamodel,dice-bundle.What's in it
metamodel so each version can be diffed against the last, and quarantines mention-types that drift
from the model for review instead of letting them silently land in the graph.
so a graph (propositions, provenance, and decay status) can be written out and reloaded faithfully
in another instance.
No pre-existing design issues map to this PR.
Closes #45
Closes #46