Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions docs/design/cross-ledger-model-enforcement.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,8 @@ The helper performs, in order:
`ctx.memo` and the global cache.
7. **Caching.** On cache hit the materialized artifact is returned
directly. On miss the artifact is inserted under the key
`(ArtifactKind, canonical_model_ledger_id, graph_iri, resolved_t)`.
`(ArtifactKind, canonical_model_ledger_id, graph_iri, head)`,
where `head` is M's `resolved_t` together with its head commit id.

A `ResolvedGraph` is term-neutral and t-fixed:

Expand Down Expand Up @@ -307,11 +308,15 @@ stabilizes — keeping the two caches separate while artifact shapes
are still evolving prevents premature coupling.

The key is `(ArtifactKind, canonical_model_ledger_id, graph_iri,
resolved_t)`. `ArtifactKind` is part of the key so a memoized
`PolicyRules` entry never short-circuits a `Shapes` lookup of the
same `(M, graph, t)`. New commits to M produce new keys without
explicit invalidation; unreferenced entries age out under the
cache's eviction policy. There is no "watermark-on-write" channel.
head)`, where `head` is M's `resolved_t` and the commit id at that
head. `ArtifactKind` is part of the key so a memoized `PolicyRules`
entry never short-circuits a `Shapes` lookup of the same
`(M, graph, head)`. `t` alone is not M's identity: a ledger dropped
and created again under M's name restarts at `t` 1, so the head
commit is what keeps it from being served the dropped ledger's
artifacts. New commits to M produce new keys without explicit
invalidation; unreferenced entries age out under the cache's eviction
policy. There is no "watermark-on-write" channel.

The cache value is the term-neutral `ResolvedGraph` (IRIs, not Sids).
Per-data-ledger interning is not part of the cache key — the cache
Expand Down
8 changes: 4 additions & 4 deletions docs/security/cross-ledger-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -492,7 +492,7 @@ Specifics:
its namespace; a constraint over vocabulary D has never seen
anywhere is silently inert (no rows, never an error).
- **Steady-state cost is one nameservice lookup.** The wire is
cached per `(model, graph, resolved_t)` and the *compiled*
cached per `(model, graph, head)` and the *compiled*
shapes (including parsed `sh:sparql` queries) are reused
across transactions while M's head and D's shape-affecting
epochs are unchanged. A commit on M invalidates both on the
Expand Down Expand Up @@ -658,9 +658,9 @@ clients can branch on the specific failure.
## Behavior on model ledger updates

There is no explicit invalidation channel. The cache key includes
the model ledger's `resolved_t` (its commit head at the time of
capture), so new commits to M produce new cache keys
automatically. The next request after M advances captures the
the model ledger's head at the time of capture, its `t` and head
commit id, so new commits to M, and a new ledger created under M's
name after a drop, produce new cache keys automatically. The next request after M advances captures the
new head; older entries age out under the cache's LRU/TinyLFU
policy.

Expand Down
9 changes: 5 additions & 4 deletions fluree-db-api/src/cross_ledger/cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
//!
//! Lives on the `Fluree` handle so resolved artifacts are shareable
//! across requests and across every data ledger that references the
//! same `(ArtifactKind, model_ledger_id, graph_iri, resolved_t)`.
//! same `(ArtifactKind, model_ledger_id, graph_iri, model head)`.
//! That is the property that makes "model edit propagates atomically
//! to every governed dataset" cheap — one cache entry is reused by
//! every D that points at the same M graph at the same M-t.
//! every D that points at the same M graph at the same M head.
//!
//! Phase 1a uses a Moka TinyLFU cache bounded by entry count. Unifying
//! with `fluree-db-binary-index::LeafletCache`'s byte budget is a
Expand All @@ -22,8 +22,9 @@
//! - Resolver: per-request memo miss → cache lookup → cache miss
//! triggers materialization → write back into both per-request memo
//! and instance cache → return.
//! - Cache invalidation is implicit: new commits to M produce new
//! `resolved_t` values and therefore new keys. Old entries age out
//! - Cache invalidation is implicit: every new head of M, including one
//! that repeats an earlier `t` (M dropped and recreated, or rebased),
//! has a new head commit and therefore a new key. Old entries age out
//! under TinyLFU eviction. There is no watermark-on-write channel.

use super::types::{ResolutionKey, ResolvedGraph};
Expand Down
8 changes: 5 additions & 3 deletions fluree-db-api/src/cross_ledger/constraints_materializer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
//! a list of property IRIs, no per-rule body, no class targets, no
//! values.

use super::types::{ConstraintsArtifactWire, WireOrigin};
use super::types::{ConstraintsArtifactWire, ModelHead, WireOrigin};
use super::ResolveCtx;
use super::{encode_system_iri, CrossLedgerError};
use fluree_db_core::{FlakeValue, IndexType, RangeMatch, RangeTest};
Expand All @@ -26,15 +26,16 @@ use fluree_vocab::config_iris;
fields(
model_ledger = canonical_model_ledger_id,
graph_iri = graph_iri,
resolved_t = resolved_t,
resolved_t = head.t,
),
)]
pub(super) async fn materialize_constraints(
canonical_model_ledger_id: &str,
graph_iri: &str,
resolved_t: i64,
head: &ModelHead,
ctx: &ResolveCtx<'_>,
) -> Result<ConstraintsArtifactWire, CrossLedgerError> {
let resolved_t = head.t;
// 1. Open M at resolved_t.
let m_db = ctx
.open_model_db(canonical_model_ledger_id, resolved_t)
Expand Down Expand Up @@ -111,6 +112,7 @@ pub(super) async fn materialize_constraints(
model_ledger_id: canonical_model_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
commit_id: head.commit_id.clone(),
},
property_iris,
})
Expand Down
5 changes: 3 additions & 2 deletions fluree-db-api/src/cross_ledger/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ pub use cache::GovernanceCache;
pub use error::CrossLedgerError;
pub use resolver::resolve_graph_ref;
pub use types::{
ArtifactKind, ConstraintsArtifactWire, GovernanceArtifact, ResolveCtx, ResolvedGraph,
RulesArtifactWire, SchemaArtifactWire, ShapesArtifactWire, WireObject, WireOrigin, WireTriple,
ArtifactKind, ConstraintsArtifactWire, GovernanceArtifact, ModelHead, ResolveCtx,
ResolvedGraph, RulesArtifactWire, SchemaArtifactWire, ShapesArtifactWire, WireObject,
WireOrigin, WireTriple,
};

/// Resolve a `f:graphSelector` IRI against a model ledger snapshot.
Expand Down
7 changes: 5 additions & 2 deletions fluree-db-api/src/cross_ledger/policy_materializer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
//! `TranslationFailed` (no silent drop; dropping a target would
//! produce a structurally weaker policy than authored).

use super::types::ModelHead;
use super::ResolveCtx;
use super::{encode_system_iri, CrossLedgerError};
use fluree_db_core::{FlakeValue, IndexType, LedgerSnapshot, RangeMatch, RangeTest, Sid};
Expand Down Expand Up @@ -74,15 +75,16 @@ use std::collections::HashSet;
fields(
model_ledger = canonical_model_ledger_id,
graph_iri = graph_iri,
resolved_t = resolved_t,
resolved_t = head.t,
),
)]
pub(super) async fn materialize_policy_rules(
canonical_model_ledger_id: &str,
graph_iri: &str,
resolved_t: i64,
head: &ModelHead,
ctx: &ResolveCtx<'_>,
) -> Result<PolicyArtifactWire, CrossLedgerError> {
let resolved_t = head.t;
// 1. Open M at resolved_t.
let m_db = ctx
.open_model_db(canonical_model_ledger_id, resolved_t)
Expand Down Expand Up @@ -234,6 +236,7 @@ pub(super) async fn materialize_policy_rules(
model_ledger_id: canonical_model_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
commit_id: head.commit_id.clone(),
},
restrictions: wire_restrictions,
})
Expand Down
66 changes: 33 additions & 33 deletions fluree-db-api/src/cross_ledger/resolver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
//! See `docs/design/cross-ledger-model-enforcement.md`.

use super::types::{
check_cycle, memo_hit, reject_if_reserved_graph, ArtifactKind, GovernanceArtifact, ResolveCtx,
ResolvedGraph,
check_cycle, memo_hit, reject_if_reserved_graph, ArtifactKind, GovernanceArtifact, ModelHead,
ResolutionKey, ResolveCtx, ResolvedGraph,
};
use super::CrossLedgerError;
use fluree_db_core::ledger_config::GraphSourceRef;
Expand All @@ -31,15 +31,15 @@ use std::sync::Arc;
/// ledger is absent or retracted on this instance.
/// 3. Reject `#config` / `#txn-meta` selectors before any storage
/// round-trip on M.
/// 4. Capture `resolved_t` lazily: read `ctx.resolved_ts[model_id]`
/// on hit, else read M's head `commit_t` once and store. Pinned
/// `f:atT` is rejected at step (1) until Phase 3.
/// 4. Capture M's head lazily: read `ctx.resolved_heads[model_id]`
/// on hit, else read M's head `commit_t` and commit id once and
/// store. Pinned `f:atT` is rejected at step (1) until Phase 3.
/// 5. Form the resolution key
/// `(ArtifactKind, canonical_model_ledger_id, graph_iri,
/// resolved_t)` and check, in order: (a) `ctx.memo` (per-request
/// `(ArtifactKind, canonical_model_ledger_id, graph_iri, head)`
/// and check, in order: (a) `ctx.memo` (per-request
/// de-dup); (b) `fluree.governance_cache()` (per-instance, shared
/// across requests and across every data ledger that references
/// the same (M, graph, t)). On hit at either layer, return —
/// the same (M, graph, head)). On hit at either layer, return —
/// cross-subsystem de-dup runs before cycle detection, and a
/// governance-cache hit is also folded into `ctx.memo` so later
/// calls in the same request short-circuit at (a).
Expand Down Expand Up @@ -141,25 +141,26 @@ pub async fn resolve_graph_ref(
// different alias spelling.
reject_if_reserved_graph(&canonical_ledger_id, graph_iri)?;

// (4) resolved_t: lazy per-request capture. f:atT pins are
// (4) M's head: lazy per-request capture. f:atT pins are
// rejected at step (1) above (Phase 3, not yet implemented), so
// every cross-ledger reference resolves against M's current head
// for the duration of the request. Per-model entries are written
// once and reused; subsequent unpinned references to the same M
// in the same request hit the cache.
let resolved_t = if let Some(t) = ctx.resolved_ts.get(&canonical_ledger_id) {
*t
} else {
let head_t = ns_record.commit_t;
ctx.resolved_ts.insert(canonical_ledger_id.clone(), head_t);
head_t
};

let key: (ArtifactKind, String, String, i64) = (
let head = ctx
.resolved_heads
.entry(canonical_ledger_id.clone())
.or_insert_with(|| ModelHead {
t: ns_record.commit_t,
commit_id: ns_record.commit_head_id.clone(),
})
.clone();

let key: ResolutionKey = (
kind,
canonical_ledger_id.clone().to_string(),
graph_iri.to_string(),
resolved_t,
head.clone(),
);

// (5a) Memo hit — short-circuit cross-subsystem de-dup before
Expand Down Expand Up @@ -188,8 +189,7 @@ pub async fn resolve_graph_ref(

// Materialize. Slice 3 provides the dispatch shape; the actual
// projector lands in slice 4.
let materialize_result =
materialize(kind, &canonical_ledger_id, graph_iri, resolved_t, ctx).await;
let materialize_result = materialize(kind, &canonical_ledger_id, graph_iri, &head, ctx).await;

// Always pop active, even on error, so a failure deeper in the
// chain doesn't leave stale entries that trip cycle detection
Expand All @@ -208,7 +208,7 @@ pub async fn resolve_graph_ref(
// write its own `Arc`. The cache ends up holding whichever
// write completed last; the loser's `Arc` is dropped when
// its request finishes. Wire artifacts are pure functions of
// `(kind, model_ledger, graph, resolved_t)`, so the two
// `(kind, model_ledger, graph, head)`, so the two
// materialized values are structurally identical — duplicate
// work, not duplicate semantics. Single-flight is a possible
// future optimization but not a correctness requirement.
Expand All @@ -225,82 +225,82 @@ async fn materialize(
kind: ArtifactKind,
canonical_ledger_id: &str,
graph_iri: &str,
resolved_t: i64,
head: &ModelHead,
ctx: &mut ResolveCtx<'_>,
) -> Result<ResolvedGraph, CrossLedgerError> {
match kind {
ArtifactKind::PolicyRules => {
let wire = super::policy_materializer::materialize_policy_rules(
canonical_ledger_id,
graph_iri,
resolved_t,
head,
ctx,
)
.await?;
Ok(ResolvedGraph {
model_ledger_id: canonical_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
resolved_t: head.t,
artifact: GovernanceArtifact::PolicyRules(wire),
})
}
ArtifactKind::Constraints => {
let wire = super::constraints_materializer::materialize_constraints(
canonical_ledger_id,
graph_iri,
resolved_t,
head,
ctx,
)
.await?;
Ok(ResolvedGraph {
model_ledger_id: canonical_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
resolved_t: head.t,
artifact: GovernanceArtifact::Constraints(wire),
})
}
ArtifactKind::SchemaClosure => {
let wire = super::schema_materializer::materialize_schema(
canonical_ledger_id,
graph_iri,
resolved_t,
head,
ctx,
)
.await?;
Ok(ResolvedGraph {
model_ledger_id: canonical_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
resolved_t: head.t,
artifact: GovernanceArtifact::SchemaClosure(wire),
})
}
ArtifactKind::Shapes => {
let wire = super::shapes_materializer::materialize_shapes(
canonical_ledger_id,
graph_iri,
resolved_t,
head,
ctx,
)
.await?;
Ok(ResolvedGraph {
model_ledger_id: canonical_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
resolved_t: head.t,
artifact: GovernanceArtifact::Shapes(wire),
})
}
ArtifactKind::Rules => {
let wire = super::rules_materializer::materialize_rules(
canonical_ledger_id,
graph_iri,
resolved_t,
head,
ctx,
)
.await?;
Ok(ResolvedGraph {
model_ledger_id: canonical_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
resolved_t: head.t,
artifact: GovernanceArtifact::Rules(wire),
})
}
Expand Down
9 changes: 6 additions & 3 deletions fluree-db-api/src/cross_ledger/rules_materializer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
//! does the same. A future variant could surface these as a warn
//! when authors mistype the rule literal.

use super::types::{RulesArtifactWire, WireOrigin};
use super::types::{ModelHead, RulesArtifactWire, WireOrigin};
use super::CrossLedgerError;
use super::ResolveCtx;
use fluree_db_core::{FlakeValue, IndexType, RangeMatch, RangeOptions, RangeTest};
Expand All @@ -27,15 +27,16 @@ use fluree_vocab::fluree::RULE;
fields(
model_ledger = canonical_model_ledger_id,
graph_iri = graph_iri,
resolved_t = resolved_t,
resolved_t = head.t,
),
)]
pub(super) async fn materialize_rules(
canonical_model_ledger_id: &str,
graph_iri: &str,
resolved_t: i64,
head: &ModelHead,
ctx: &ResolveCtx<'_>,
) -> Result<RulesArtifactWire, CrossLedgerError> {
let resolved_t = head.t;
let m_db = ctx
.open_model_db(canonical_model_ledger_id, resolved_t)
.await
Expand All @@ -62,6 +63,7 @@ pub(super) async fn materialize_rules(
model_ledger_id: canonical_model_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
commit_id: head.commit_id.clone(),
},
rules: Vec::new(),
});
Expand Down Expand Up @@ -99,6 +101,7 @@ pub(super) async fn materialize_rules(
model_ledger_id: canonical_model_ledger_id.to_string(),
graph_iri: graph_iri.to_string(),
resolved_t,
commit_id: head.commit_id.clone(),
},
rules,
})
Expand Down
Loading
Loading