Skip to content

fix: JSON-LD graph scoping, config integrity and explicit-only SHACL - #2009

Open
aaj3f wants to merge 64 commits into
mainfrom
fix/jsonld-graph-scope-config-integrity
Open

aaj3f wants to merge 64 commits into
mainfrom
fix/jsonld-graph-scope-config-integrity

Conversation

@aaj3f

@aaj3f aaj3f commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

This is deliberately more than #1979 needs. Which graph a write lands in and which checks run on it were decided per JSON-LD keyword and per write lane, and the answers drifted: nested nodes lost their graph (#1979), named-graph content was silently dropped, config groups split across graphs, and a heuristic decided whether SHACL ran. So rather than patch one surface, each answer now has one home, which is easier to reason about and tune: one rule for a node's graph, one check path shared by the write lanes instead of three drifted copies, and SHACL exactly where config turns it on. Fwiw, it's one of five PRs taking this approach: #2006 and #2008 have merged, and #2007 and #2010 are open.

Follows #1979, which #2022 closed. #2022's 7d894b7e9 makes a node's own @graph the default graph for the node objects nested in it, and 4a4f7c309 names the construct a TriG GRAPH block refuses. This PR makes the first fix the one rule for every JSON-LD reader, and covers the config half of #1979 (item 2, the config docs, and the checks behind them).

Fixes #1991: an annotated edge under ["graph", …] or a top-level graph is written, and deleted, with its annotation in that graph.

Before #2022, a JSON-LD config written the documented way (docs/ledger-config/writing-config.md) silently did not work. The config node landed in #config, but its nested f:shaclDefaults group landed in the default graph, so the reader saw an empty group and SHACL stayed off, while a ledger with shapes and no config enforced them anyway. #2022 fixed where the nested group lands. This PR closes the class around both halves: B1 makes a JSON-LD template's graph its innermost enclosing lexical scope, in every reader, B2 checks config integrity and makes SHACL explicit-only, and B3 gives an IRI an update binds one encoding wherever it binds it.

Heads-up for reviewers: B2 is a breaking change (ledgers that hold shapes without f:shaclEnabled true stop enforcing on upgrade), so the behavior-change list and the enable recipe below are the parts I'd most like eyes on.

What changed

B1: JSON-LD graph scoping. A template's graph is its innermost enclosing lexical scope, in every JSON-LD reader:

  • Nested nodes (and @list items, anonymous nodes, @reverse / @included / @nest content) are written to their enclosing node's graph. A nested node's own @graph wins for its subtree; "default" names the default graph.
  • JSON-LD 1.1 named graph objects ({"@id": G, "@graph": [...]}) are supported. Named-graph object content that was silently dropped is now stored.
  • One @graph classifier and one graph-name resolution order (classify_written_graph_name), shared by templates and where: a variable, a fromNamed alias, the keywords "default", "config" (the ledger's own config graph) and "txn-meta", then expansion against the @context (absolute and compact IRIs; @vocab does not apply). Relative graph names are refused. A ?var graph name in an update template writes to the graph the WHERE binds (SPARQL GRAPH ?g parity); elsewhere it is refused.
  • Blank node labels are scoped to the whole document (one issuer across ["graph", …] items; user labels never collide with minted ones).
  • Only envelope documents carry transaction metadata; a single object with a string @graph selector does not. At the top level, a document without @id whose @graph is a single node object is read exactly like the one-element array: envelope content, a bare reference {"@id": X} included. (Solo MCP's single-node envelope {"@graph": {…}, "f:message": …} keeps its metadata.)
  • Edge annotations follow their edge into its graph, with one f:reifiesGraph emitter (GraphScope::emit). An annotation bundle is placed in its edge subject's @included, so it lands in the edge's graph by construction (no run-time cross-check, and nothing for RDF 1.2 triple terms: rdf:reifies links, triple terms as values, faster annotation queries #2017's reifier links to conflict with). One edge-key classifier, shared by the insert and delete walkers, tells a keyword, a reverse-defined term and a forward term apart, so an annotation on a reverse property reifies the edge as stored. Deleting an annotated edge addresses the edge's graph, not the object's, and the selector form resolves its graph name as a template does. An annotated edge under a top-level graph naming the default graph (the ledger's own address, fix: unresolvable graph references fail closed, and TriG directives apply in document order #1997, or urn:default, fix(query): 4xx for unknown graphs and commit refs; urn:default names the default graph #2005) lands with its annotation in the default graph. An annotation under a variable graph name is refused while parsing.
  • A node-level @graph in a JSON-LD where is a GRAPH pattern, and its name resolves in the order above, so an update whose where and delete both say "@graph": "ex:g" address one graph. In an update, "default" there names the ledger's default graph, as in the templates and as urn:default does, also when a top-level graph or from gives the where another default graph, so moving data from the default graph into a named graph reads what it deletes; an update's where never reads a union default graph. A nested node's own @graph scopes its patterns; "default" inside an enclosing graph pattern is refused (a where pattern cannot leave it). A ["query", …] subquery names graphs as the where around it does. Each name in an update's fromNamed (a graph's IRI, its alias, the name it is listed by) must mean one graph across the list, as a query's dataset aliases must, and "default" in a from list reads the default graph beside the graphs listed (it was dropped). Top-level and nested where nodes read their @graph selector through one path.
  • In a CONSTRUCT template, a node's own @graph names the graph the node's triples are written to, as ["graph", …] and SPARQL's GRAPH block do. The "construct": true shorthand, which uses the where clause as the template, keeps a where node's own @graph the same way (SPARQL's CONSTRUCT WHERE has no GRAPH form and is unchanged).
  • @reverse, @included and @nest are implemented; @index is ignored; other keyword-form (^@[A-Za-z]+$) keys on a node are refused. Non-keyword @ keys such as @odata.etag behave exactly as before.
  • Bulk JSON-LD import refuses @graph scoping instead of folding it into the default graph.
  • A scoped (property- or type-scoped) context's term IRIs written as compact IRIs over a prefix only the outer context defines ("a": {"@context": {"b": {"@id": "ex:b"}}}) expand against the context where the scoped context applies; they stayed compact, so the key b became the string ex:b, which a transaction stored as an IRI. This predates the PR.
  • JSON-LD expansion descends a level without allocating (its position in the document is a chain of borrowed steps, collected only to report a nested array) and borrows the active context instead of cloning it for every node. Parsing a document of 2,000 annotated edges takes 28–36% less time than on main. That speedup is this change's, independent of the annotation placement: against a main with the same change, default-graph annotation parse is +10.3% and an end-to-end insert −2.0% (Performance).

B2: config integrity and explicit-only SHACL.

  • SHACL is enforced only where the ledger config enables it (f:shaclEnabled true, ledger-wide or per graph). The shapes-exist heuristic is gone; posture is a pure function of the resolved config, on every lane (JSON-LD/SPARQL, Turtle insert, branch merge/rebase/revert, push).
  • One post-staging check tail for every write lane (check_staged_write): one config read, then SHACL, uniqueness and the staged-config guard (every check on authoring lanes; the config-group re-check on branch operations). It replaces three copies that had drifted: the plain Turtle insert lane now enforces uniqueness like the other lanes and applies the caller's inline shapes, requested validation mode and inline uniqueness constraints, which a plain Turtle document ignored while TriG honored them, and merge, rebase and revert (and their previews) enforce uniqueness as well as SHACL. Push is the one exception: it validates pushed commits through its own SHACL call, with replay semantics (origin_validated_replay; a violation is a 422), and runs neither uniqueness nor the config-group checks. Routing it through the tail would change its statuses and add those checks to replication, so it is a follow-up (Write paths outside the shared check tail: push, validate, and stage + commit_staged #2076). Push takes graph ids from the ledger's graph registry (the registered id, or the id applying the pushed commits assigns), so each named graph a push writes is validated under its own config; a pushed flake in a graph that neither the ledger nor the pushed commits register is refused (422) before anything is written.
  • Fast-forward merges into a governed target are validated. A fast-forward adopted the source's commits unvalidated, as "validated when they were authored". Nothing guarantees that commits authored on another branch satisfy the target's configuration, so a fast-forward into a target whose config enables SHACL or uniqueness (ledger-wide or for any graph) now stages the adopted change on the target and runs the transaction checks under the target's config as it was before the merge, before any commit is copied; the merge preview reports the outcome. A fast-forward whose adopted commits carry config settings is checked the same way (next bullet). Any other fast-forward is unchanged.
  • Branch operations re-check the target's config groups. Merges (fast-forwards included), rebases and reverts run the staged-config guard on what they stage. Config typed outside the config graph is refused when the data is written; branch operations re-check the integrity of the target's config groups (no group split across graphs, within the change and across transactions, and no second value for a single-valued setting), so a merge cannot combine a group link and the group's fields outside the config graph that were each accepted where they were written, and a revert cannot undo a repair. The refusal is a conflict with the target's state (409, BranchConflict), naming the target's config group and pointing at the repair recipe; previews report it, and a rebase stopped by one names the commit it was replaying. A merge or rebase that would carry what a branch wrote to its own config or txn-meta graph (urn:fluree:<ledger>:<branch>#config) into another branch is refused the same way: those graphs configure the branch that wrote them, and on the target they would be ordinary graphs. Merging from a branch the target was created from is unaffected. The other direction is refused too: a merge (fast-forward or not) that would change the target's own config or txn-meta graph with what a branch wrote there, where that IRI names an ordinary graph, is a 409. A branch's config is changed on that branch.
  • Inline request shapes (opts.shapes) validate on their own pass, gated by the SHACL group's f:overrideControl rather than f:shaclEnabled; where override control refuses them the transaction fails with a 400 (RequestOverrideRefused). They are not supported in a policy-scoped request (refused with a 400), and neither are inline uniqueness constraints (opts.uniqueProperties); support for both in policy-scoped requests is a follow-up. opts.shapes and opts.uniqueProperties in a transaction body are read on every surface (HTTP, the embedded API and the CLI's local mode; only the HTTP route read them), malformed values are refused, and a build without SHACL support refuses inline shapes.
  • GraphQL: sh:minCount ≥ 1 makes a field non-null only where the ledger config enables SHACL for the view's graph. Elsewhere nothing enforces it, so the field is nullable.
  • Validation artifacts (shapes, schema and constraints sources, including cross-ledger model ledgers) are resolved only for graphs that participate. The system graphs never participate in SHACL or uniqueness, so a config-only write is never blocked by an unresolvable source and the owner can always repair the config. Policy sources are unchanged: they still fail every write closed, config writes included.
  • Config reads on the transaction path fail closed uniformly (uniqueness used to read an unreadable config as "no config"); a config-only write reads none. A failure to read the config is its own error with one status (HTTP 500, err:storage/ReadFailure, in fix(server): one HTTP status table for API errors #2102's one status table) on the write lanes, branch operations, push and queries: TransactError::ConfigUnavailable on every staging lane (transactions, multi-operation updates, branch operations, push), and ApiError::ConfigUnavailable for the transaction policy context, f:policySource resolution and a query's config. It was a Parse error (400) when staging, a 422 on push, and Invalid configuration (400) elsewhere.
  • A multi-operation request is governed by the state before it. Each operation of a multi-operation SPARQL update, each clause of a Cypher write, and each write staged in a transaction (Fluree::begin_transaction, feat(api): multi-write transactions #2023), a nested multi-operation update or Cypher statement included, resolves SHACL, uniqueness and config from the state before the request, as one transaction does, in both directions: an earlier operation that turns enforcement off, or removes a shape or constraint, does not admit a later operation's data, and one that turns enforcement on, or adds a shape or constraint, does not apply it to the same request. A model ledger the config names (a cross-ledger f:shapesSource) is read at one head for the whole request, the one its first lookup found (commit 61, on fix(api): key cross-ledger governance by the model ledger's head commit #2052's head pins). So a request main refused can now commit: a shape and data violating it, written in one multi-operation update, both commit. The config checks run once, on the request's result, so INSERT DATA {…} ; DELETE DATA {…} is judged by what it commits, not by operation order. Explicit Bolt transactions are not requests of this kind: they commit each statement as it runs, so each statement is governed by the state before it, as separate transactions are.
  • A write staged before a uniqueness declaration stages again. A commit that declares a property unique (f:enforceUnique) now makes a write or transaction staged before it stage again when it commits, as a commit that changes the shapes, the class hierarchy or a named graph already did, so the write is checked against the declaration rather than re-based over it.
  • The config reader treats an empty group as absent and resolves a doubled group pointer deterministically. Ledger info reports degenerate config as configDiagnostics (shacl-not-configured, empty-group, stranded-fields, ambiguous-pointer, multiple-ledger-configs, duplicate-graph-override); fluree info prints them, reading them through config_resolver::diagnose rather than the full ledger info, and MCP's ledger-info output lists them. A shape counts for shacl-not-configured whether it is typed or only has a SHACL target (sh:targetClass, sh:targetNode, sh:targetSubjectsOf, sh:targetObjectsOf), and a config that sets f:shaclEnabled false is not reported. The query group main gained (f:queryDefaults, feat(query): union default graph, per ledger and per query #2004) is read, diagnosed and guarded like the others, and a test destructures LedgerConfig field by field: a setting group added later breaks the test build until it is mapped to its group pointer, and the test then fails until the guard and the diagnostics list that pointer. A second test reads the config vocabulary's source and requires each of its terms to be a guarded field or listed as a class, a value or a data term, so a setting added inside a group fails it too. The config vocabulary is one module (config_vocab.rs): the guard, the diagnostics, graphs_with_config_fields, carries_config_settings and the repair recipe derive from it, a test records every predicate the reader looks up while it resolves a maximal config and requires the vocabulary to equal it, and another parses the documented recipe and requires it to follow every config edge. Diagnostics walk every node reachable from the config root, so a stranded per-graph override, a partly stranded group and an override without a target are reported, and so are config fields on the recommended config subject when the config graph does not hold it; the reader's two per-resolve warnings are throttled. fluree info prints the diagnostics for a tracked ledger too. A ledger that holds shapes no config enforces logs that once each time it loads (reading the ledger only when its index stats or novelty show a shape type or a target predicate).
  • A staged-config guard refuses (as Parse error, HTTP 400) a setting group split across graphs (the root f:LedgerConfig subject's own fields included), a config typed outside the ledger's own config graph, config fields written outside it on the recommended config subject urn:fluree:<ledger>:config:ledger (also on a ledger with no config, where that IRI is what marks the write as config), and a second value for a single-valued setting or a second f:LedgerConfig subject. The split check holds across transactions too, in either order: config fields (the predicates the config reader reads) written outside the config graph for a group the config graph already points at, and a config edge to a node whose config fields sit in another graph. Other f: data in a user graph (an edge annotation's f:reifies*, a property's f:enforceUnique, a policy document's f:allow) is not a config field, and writing it reads no config. Moving the fields in with the edge in one transaction (the repair) is accepted.

B3: one encoding for an IRI an update binds (commits 28 and 29).

  • Every IRI an update writes is stored under its canonical namespace code, whichever path bound it (commit 28). A where constant in a namespace the ledger has not seen was stored under the encoder's fallback code; once the namespace got its code, a lookup by the IRI missed what that write stored, as a subject and as an object, in JSON-LD and SPARQL updates alike. Facts already stored that way stay as they are: there is no migration.
  • An update's where reads every IRI in the ledger's encoding (commit 29). An IRI the update's top-level values binds, in a namespace the ledger has not seen, reached the where with the code the transaction gave that namespace, which the where could not decode: STR gave the encoded form ("15:thing"), which an insert then stored, and GRAPH ?g bound to urn:default read nothing. It now enters as the same IRI in a where VALUES does, and a write stores it under its canonical code.

Behavior changes (release note)

Suggested label: breaking-change. The full text is in docs/reference/compatibility.md#behavior-changes.

  • Breaking: SHACL is enforced only where the ledger config enables it. Ledgers that hold shapes without f:shaclEnabled true (including shapes published through GraphQL or fluree model entity define) stop enforcing on upgrade; merge, rebase, revert and push stop validating on them too. Configs without a SHACL group were already unenforced and do not change. Ledger info reports affected ledgers as shapes present; SHACL enforcement not configured, and such a ledger logs it once each time it loads. To keep enforcing, enable SHACL in the config (recipes below): for a ledger that holds shapes and no config, one INSERT DATA. Ledger info counts untyped shapes (a subject with a SHACL target) too, and does not report a config that sets f:shaclEnabled false. A branch created from a ledger that has no index yet reads none of its source's config (A branch of a never-indexed ledger ignores its source's ledger config: configured policy fails open, uniqueness is off #2069, which predates this PR): the shapes-exist fallback still enforced SHACL on such a branch, and with this change SHACL is off there until A branch of a never-indexed ledger ignores its source's ledger config: configured policy fails open, uniqueness is off #2069 is fixed.
  • Fast-forward merges into a target whose config enables SHACL or uniqueness are validated against the target's config as it was before the merge (a merge whose commits also change that config is judged by the config it merges into). A violating fast-forward is refused like a general merge, and fluree branch diff / GET /merge-preview report its validation. Fast-forwards into other targets are still not validated against shapes or uniqueness.
  • Merge, rebase and revert enforce uniqueness constraints as well as SHACL, and the merge and revert previews report a violation. A plain Turtle insert enforces uniqueness like the other lanes, and applies programmatic inline shapes, a requested validation mode and inline uniqueness constraints (TxnOpts), which it ignored.
  • Merges (fast-forwards included), rebases and reverts refuse a result that would leave one of the target's setting groups split across graphs or a single-valued setting with two values, as a conflict (409). A branch that carries a split config from before this release is refused until the config is repaired (recipe below). A rebase refused this way names the commit it stopped on.
  • A merge or rebase that would carry what a branch wrote to its own config or txn-meta graph into another branch is refused (409). Merging from a branch the target was created from is unaffected. A merge (fast-forward or not) that would change the target's own config or txn-meta graph with what another branch wrote there is refused (409) as well.
  • Each operation of a multi-operation SPARQL update, each clause of a Cypher write, and each write staged in a transaction is checked against the shapes, uniqueness constraints and config in effect before the request, and reads a cross-ledger model ledger at one head; the config checks run once, on the result. Requests whose earlier operations relaxed enforcement for later ones are refused, and requests whose earlier operations tightened it now commit (a shape and data violating it, in one multi-operation update; main refused them). Nothing a single operation accepts changes. Explicit Bolt transactions are unchanged: each statement commits on its own, governed by the state before it.
  • A write or transaction staged before another commit that declares a property unique stages again when it commits, and is checked against the declaration.
  • Push validates each named graph a push writes under its own config, and refuses (422) a pushed commit that writes a graph which neither the receiving ledger nor the pushed commits register.
  • A failure to read the ledger config is HTTP 500, err:storage/ReadFailure, on every lane: transactions, multi-operation updates, branch operations, push, the transaction policy context and queries (a tracked query's answer carries only the status, as fix(server): one HTTP status table for API errors #2102 notes for every tracked error). It was a 400 Parse error when staging, a 422 on push, and a 400 err:system/ConfigError for the policy context and queries.
  • JSON-LD nodes nested under a node with an @graph selector are written to that graph (in v4.2.3 they landed in the default graph; main has done this for nested node objects since fix: engine fixes found building the Python package #2022, and this PR applies the rule to every reader and node shape). A client that relied on the old placement should give the nested node "@graph": "default".
  • Named-graph object content that was silently dropped is now stored.
  • Only an envelope's top-level keys are transaction metadata: a single node with an @graph selector, and a named-graph object's own keys, are data only (they were written as data and as metadata). A top-level {"@graph": {…}} without @id stays an envelope, as before.
  • A node-level @graph in a JSON-LD where is a GRAPH pattern, with the graph name resolved as in a template (it was read as a predicate named @graph and matched nothing). In an update, "default" there names the ledger's default graph, as in the templates and as urn:default does, even under a top-level graph or from, and an update's where never reads a union default graph. In a query it names the query's default graph, which a union default graph widens; urn:default names the default graph alone. A fromNamed list that gives one name to two graphs is refused (Parse error, 400), an alias spelled urn:default that names another graph among them; "default" in an update's from list now reads the default graph (it was dropped); a ["query", …] subquery names graphs as its where does; in a CONSTRUCT template, the "construct": true shorthand included, a node's own @graph names the graph its triples are written to (it was read as a property named @graph, and the triples went to the default graph); and opts.unionDefaultGraph is refused on a JSON-LD transaction, as SPARQL refuses its pragma on an update.
  • Relative graph names are refused, and a graph name no longer expands through @vocab (such a name is refused as relative); "config" names the ledger's config graph.
  • An update reads an IRI its top-level values binds as written, also in a namespace the ledger has not seen: STR gave the encoded form and an insert stored it, and GRAPH ?g read nothing for urn:default. An IRI an update's where binds in such a namespace is stored under its canonical code, so it is found by IRI with the ledger's other facts about it; facts already stored under the fallback encoding stay split, with no migration.
  • Inline request shapes follow override control (they now apply even under f:shaclEnabled false, and are refused under f:OverrideNone); they are refused in policy-scoped requests, as are inline uniqueness constraints (a request with an identity is policy-scoped, so opts.shapes sent with an identity is a 400). opts.shapes and opts.uniqueProperties in a transaction body are honored by the embedded API and the CLI's local mode (they were ignored there), and malformed values are refused.
  • Config writes are checked (the guard above), including a setting group split across two transactions, and config fields written outside the config graph on the recommended config subject, also on a ledger with no config. A config-only write is never validated against shapes or uniqueness constraints (shapes targeting config nodes are no longer checked).
  • Unknown keyword-form @ keys on a node are refused; @nest is honored; @index is ignored. Edge annotations under a variable graph name are refused while parsing (they failed at commit with an unhelpful message).
  • Bulk JSON-LD import refuses @graph scoping; use fluree insert, or TriG / N-Quads.
  • A context term defined with "@reverse" writes its edge in reverse (object → subject), as JSON-LD defines it; it wrote it forward. Data written that way stays as it is (below). @annotation on a value of a reverse property annotates the edge as stored.
  • GraphQL: sh:minCount ≥ 1 makes a field non-null only where the ledger config enables SHACL; elsewhere the field is nullable.
  • A scoped context's term IRIs over an outer prefix ({"b": {"@id": "ex:b"}} under an outer ex) now expand to full IRIs; such keys were written as the compact string (ex:b) and stored as an IRI with the scheme ex. Data already stored that way stays as it is.
  • Public API: fluree_db_transact::parse_transaction takes the target ledger's id as a new last argument; graph names resolve against it ("config", and the ledger's own reserved graph IRIs). Solo does not call it.
  • Data written by earlier releases (no format change, no migration):
    • a config with two values for one setting-group pointer now reads the non-empty group, so a SHACL or policy group that had no effect takes effect on upgrade, a stored f:defaultAllow false included; diagnostics flag it as ambiguous-pointer;
    • edges written through a reverse-defined term ("childOf": {"@reverse": "ex:child"}) were stored as if the term were not reversed, so the same document no longer reaches them (a delete with it does nothing); docs/reference/compatibility.md gives a repair query.

Enable SHACL on a ledger that holds shapes and no config (the usual affected ledger):

PREFIX f: <https://ns.flur.ee/db#>
INSERT DATA {
  GRAPH <urn:fluree:mydb:main#config> {
    <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
      f:shaclDefaults <urn:fluree:mydb:main:config:shacl> .
    <urn:fluree:mydb:main:config:shacl> f:shaclEnabled true .
  }
}

On a ledger that already has a config subject, attach the group to it instead (a second f:LedgerConfig subject is refused; this form does nothing on a ledger without config):

PREFIX f: <https://ns.flur.ee/db#>
INSERT {
  GRAPH <urn:fluree:mydb:main#config> {
    ?c f:shaclDefaults <urn:fluree:mydb:main:config:shacl> .
    <urn:fluree:mydb:main:config:shacl> f:shaclEnabled true .
  }
}
WHERE { GRAPH <urn:fluree:mydb:main#config> { ?c a f:LedgerConfig } }

Repair a config split by #1979 (policy and SHACL groups it affected: a split group's settings, f:defaultAllow false included, aren't in effect until it's repaired; merges, rebases and reverts that carry a split config are refused until it is repaired). It starts from the config subject and from every node a config edge in the config graph points at, and follows config edges through the default graph. Run with SELECT ?n ?p ?o first:

PREFIX f: <https://ns.flur.ee/db#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

DELETE { ?n ?p ?o }
INSERT { GRAPH <urn:fluree:mydb:main#config> { ?n ?p ?o } }
WHERE {
  {
    GRAPH <urn:fluree:mydb:main#config> { ?root a f:LedgerConfig }
  } UNION {
    GRAPH <urn:fluree:mydb:main#config> { ?parent ?edge ?root }
    VALUES ?edge { f:policyDefaults f:shaclDefaults f:reasoningDefaults f:datalogDefaults
                   f:transactDefaults f:fullTextDefaults f:servingDefaults f:queryDefaults
                   f:graphOverrides f:overrideControl f:shapesSource f:policySource
                   f:schemaSource f:rulesSource f:constraintsSource f:graphSource
                   f:trustPolicy f:rollbackGuard f:ontologyImportMap f:graphRef f:property }
  }
  ?root (f:policyDefaults|f:shaclDefaults|f:reasoningDefaults|f:datalogDefaults|
         f:transactDefaults|f:fullTextDefaults|f:servingDefaults|f:queryDefaults|
         f:graphOverrides|f:overrideControl|f:shapesSource|f:policySource|f:schemaSource|
         f:rulesSource|f:constraintsSource|f:graphSource|f:trustPolicy|f:rollbackGuard|
         f:ontologyImportMap|f:graphRef|f:property|f:reasoningModes|rdf:rest)* ?n .
  ?n ?p ?o .
}

The recipes are exercised by tests: enable_recipe_enables_shacl_with_or_without_an_existing_config; repair_recipe_restores_a_split_config, which runs the docs' recipe verbatim on four split shapes (a nested group, the root subject's own link, a stranded per-graph override, a partly stranded group); and the_repair_recipe_follows_every_config_edge, which parses it and checks it follows every config edge the reader reads.

Solo lockstep

From an audit of fluree/solo's write paths, updated for the review changes:

  • Branch merges (agent writes through review). With agents writing to <dataset>:<branch> and a person merging, a merge into an idle main is a fast-forward. When main's config enables SHACL or uniqueness, that fast-forward is now validated against main's config and refused on a violation, with the errors a general merge gives; the merge preview reports it as validation (mergeable false). Solo's review UI should show a fast-forward preview's validation too. Merge, rebase and revert also refuse uniqueness violations now, and a result that would split one of the target's config groups (a 409 naming the group); a fast-forward whose commits carry config settings is checked that way even into an ungoverned main. A merge that would carry what an agent branch wrote to its own config or txn-meta graph into main, or change main's config or txn-meta graph with what the branch wrote there, is a 409 too, so config changes are made on main itself. Solo writes a dataset's config under its branch-qualified IRI (fluree-lambda-model's config_graph_iri), so neither refusal reaches its writes as they are today.
  • SHACL posture UI. Solo's readPosture renders a ledger with no stored f:validationMode as "enforce" (readPosture, graphs.ts:262; QualityTab.tsx:152, solo main at c1efea7c6). For ungoverned ledgers that hold their own shapes and no config (user imports, ledgers unbound with governanceTx(…, null)), that label becomes false: return a third "not enforced" state when no enabled SHACL group exists, or write an explicit posture at import/unbind. The MCP tools' validationMode: "reject" does not turn SHACL on for such a ledger either (the heuristic did). Governed ledgers already set f:shaclEnabled true and are unaffected.
  • Config guard: posture writes (needs a solo change). On a model ledger, the model Lambda's fulltext setup creates urn:fluree:<ledger>:main:config:ledger a f:LedgerConfig, and the web posture toggle (postureTx, governance.ts:249, sent by graphs.ts:249 from ModelHead.tsx and DatasetSettings.tsx) then inserts its own urn:cfg:main a f:LedgerConfig. This PR refuses that second config subject (400). On main it commits, and the reader silently keeps urn:cfg:main and drops the fulltext config. Solo should attach the posture to the existing config subject (the docs' INSERT … WHERE form, falling back to INSERT DATA when there is none); governanceNodes has the same shape. This needs to land before or with the pin bump. Latent refusal: a re-run of the model lambda's ensure-fulltext-config would add a second f:fullTextDefaults group (make it idempotent). The refusal of config fields on the recommended config subject outside the config graph covers the f: config vocabulary only: the ftm:coreLanguages set solo keeps on that node in the default graph is data and is unaffected.
  • JSON-LD writers. Named-graph objects from the spaCy extractor (…:knowledge) and semantic-transform model_jsonld (urn:fluree:workflow:vocabulary:<id>) now store their content in that named graph instead of dropping it; solo's default-graph readers and cleanup do not see it, so emit an envelope or strip the wrapper @id as pack-ner and apply-schema do. MCP stamp_provenance on an agent document with a top-level @id and an @graph (a named graph object, or a node with a graph selector) makes the stamped keys data of that node rather than transaction metadata. On a context-free bare reference ({"@id": …}, wrapped as {"@graph": {"@id": …}, …}) it is refused, as before. MCP jsonld_update: a where node's "@graph" now selects that graph, with compact names expanded against the document's @context and "default" naming the ledger's default graph (it matched nothing before). Bulk import of JSON-LD with @graph scoping now fails with the engine message instead of folding the data. @odata.* keys behave as before; custom webhooks sending @nest now merge it.
  • Catch-up with main (2941d470c). Nothing new reaches solo: it names urn:default only to label the default graph in its graph list (GraphDetail.tsx), uses fromNamed only in query datasets (the new refusal is for an update's fromNamed alias), and writes no f:queryDefaults group (code search of solo's default branch, Oct 5). The later review changes reach no solo call either: solo sends no opts.unionDefaultGraph, no update fromNamed and no from list holding "default", and commits 28 and 29 change only what an update reads and stores for an IRI in a namespace the ledger has not seen.
  • Errors. Every new refusal on a transaction renders as Parse error: , JSON-LD error: or Unsupported feature: (all mapped to 400 by solo's router), except RequestOverrideRefused for inline shapes, which solo does not send, and a named-graph object whose @id is the ledger's #txn-meta graph: staging's existing reserved-graph check refuses it as transaction targets reserved system graph …, which solo's router maps to 502 (map it to 400). A branch operation the config checks refuse is now a 409 (it was 400 Parse error). A config read failure is HTTP 500, err:storage/ReadFailure, on every lane. To retry it like other storage failures, solo needs two arms:
    • The transact Lambda's classify() treats every ApiError::Transact(_) as deterministic before it looks at the status, so TransactError::ConfigUnavailable goes in its transient arm. Without it, solo behaves as today.
    • The new SubmissionError::ConfigUnavailable keeps the code through the consensus flattening. The router's exhaustive classify_submission() needs an arm for it, as transient, or the pin bump does not compile.

Sequencing with bplatz's engine PRs

This branch is rebased on main at c5874d586, which carries #2022–#2025, #2006, #2008, #2013, #2036 and #1451:

Performance

Across two criterion rounds on a shared box, the JSON-LD parse and insert benches move a few percent in both directions (the untouched Turtle bulk-import path moved −10% to +18%, which is the noise floor here), and jsonld_parse's update_sugar case is 22% faster. jsonld_parse annotations went over the 5% budget twice. It is now faster than main only because of the expander fix (commit 33), which is independent of the annotation placement and would speed up main by itself: against a main with the same fix, default-graph annotation parse is +10.3% and an end-to-end insert −2.0% (Annotation parse, below). These criterion numbers predate commits 17–27 and use default features, so they don't exercise the SHACL transaction path this PR restructures; the quiet box below and the per-commit counts in the performance comment do. All of these measurements predate the catch-up with main at 2941d470c, which adds no work to the measured insert and commit paths: commit 27's change is in an update's where (see its bullet in the performance comment), the query group is one more entry in the guard's field set (a hash set), and the reader reads that group with the same two lookups main makes. They also predate the rebase onto c5874d586 and commits 48–58, of which only commit 50 adds work on a path every ledger takes: the load-time check also looks for SHACL target predicates, in the index's property statistics and in novelty. Measured on cold loads against the commit before it (8 alternating rounds), that is +0.6% instructions on an indexed ledger without shapes (wall time within noise) and nothing measurable unindexed; an indexed ledger whose only shapes are untyped now pays the warning's lookup, as one with typed shapes already did (the performance comment has the table).

Annotation parse. The first overrun (+9.9% / +7.8%) was the reifier cross-check allocating a key per template; it was made to borrow its keys (+2.2%), and later removed when annotation bundles moved into their edge subject's @included (commit 34). That placement was the second: it nests each bundle two levels deeper, and the JSON-LD expander copied its position in the document (a vector of keys and indexes, one allocation per key) for every key and item it descended into, and cloned the active context for every node. Commit 33 makes descending allocation-free and borrows the context. That fix is independent of the placement: on main alone it takes 35% off this parse and 11–12% off the insert, so the speedup against main in the table is the fix's, not the placement's. This PR's own cost is the last column, against a main with the same fix: +10.3% on the default-graph parse, −2.0% on the end-to-end insert. A harness (not committed; dev-fast with shacl, Apple M4) parses 2,000 annotated edges 20 times per round, or inserts them into a fresh memory ledger 5 times, with four builds rotated over 9 rounds on a shared machine (load 9-15). "Fixed main" is main with the same expander change, to separate this PR's own cost from the expander's saving. Median of the per-round means:

Case main before commit 33 this head vs main vs fixed main
parse, default graph 18.43 ms 20.18 ms 13.26 ms −28.1% +10.3%
parse, under a node @graph 23.20 ms 21.62 ms 14.81 ms −36.2% −1.9%
insert, default graph 40.28 ms 41.45 ms 34.99 ms −13.1% −2.0%
insert, under a node @graph 49.99 ms 48.71 ms 41.76 ms −16.5% −4.9%

Against main the head was faster in every round on every row. Against fixed main it was slower in 9 of 9 rounds on the default-graph parse, and in 2, 3 and 2 of 9 on the other three rows. Against fixed main, the default-graph parse keeps +10%: about half is the placement itself (+4.8% against the commit before it, both with the expander change, in a second run of 10 rounds), and half comes from the commits before it (a profile puts most of it in annotation lowering under the graph scope). End to end, the insert, it is at parity or better. Sibling placement for a subject that sits in the root array would bring that parse row to about +5%, at the cost of a second placement path; it was not built.

bplatz's review changes, measured against main (6b9d5b619, before the last rebase) and this branch before those changes. A harness (not committed; dev-fast with shacl, Apple M4, data on the internal disk) runs each write lane on a memory ledger with no config and on a governed one (SHACL enabled with a shape, plus a uniqueness constraint): 20 warm-up operations, then 300 timed, three builds rotated per round. Instructions retired per operation, median of 6 processes per build:

Lane main before this head vs main vs before
insert, 1 node, no config 615,395 564,120 564,700 -8.2% +0.1%
insert, 1 node, governed 2,294,724 1,850,591 1,849,231 -19.4% -0.1%
upsert, 1 node, no config 1,613,745 1,562,522 1,561,646 -3.2% -0.1%
upsert, 1 node, governed 2,909,054 2,463,178 2,463,943 -15.3% +0.0%
update with a where, 1 subject, no config 753,816 705,927 705,958 -6.3% +0.0%
update with a where, 1 subject, governed 2,040,114 1,599,699 1,597,702 -21.7% -0.1%
two-operation SPARQL update, no config 1,707,824 1,606,060 1,578,771 -7.6% -1.7%
two-operation SPARQL update, governed 3,579,258 2,695,882 2,245,186 -37.3% -16.7%

On a single-operation write these changes are within ±0.1% of the branch before them. A multi-operation request reads its governance once, from the state before it, instead of once per operation (−16.7% governed). The gap to main comes from rounds 1–3 (the check tail, the config read and the explicit-only posture); it was not split commit by commit here. Re-checked after the rebase onto 745736a04 (6 alternating processes per build): this head against that main is −3.1% to −36.8% on every lane, and a two-operation Transaction (#2023: a JSON-LD insert and a SPARQL update, committed) is −5.3% with no config and −32.2% governed. Part of the governed saving is the one every governed single-operation lane shows against main (−15% to −22%, from the check tail and the explicit-only posture); the rest is reading governance once per transaction rather than once per operation.

Wall time for push, cold loads and fast-forwards, the instruction counts for commits 28 and 29, the commit cost with shacl, the fast-forward merge timings and the original criterion table are in the performance comment below. In short: push is unchanged; the load-time warning costs +76 to +95 µs on an indexed ledger that holds shapes without config; a fast-forward into a cached target pays +0 to +2.5 ms, and one on a fresh instance pays the target load (on an unindexed target of 1,000 commits, 166.7 ms against 16.3 ms on main); a commit does less work than on main (−11% instructions on a fresh ledger).

Later changes. The restage on a uniqueness declaration (commit 43) adds one comparison per staged flake on commit. Instructions per commit through the cached-handle commit path, before and after, 8 alternating processes (dev-fast, default features): 720,808 → 720,746 for a 1-node insert (−0.01%) and 17,353,578 → 17,365,012 for a 100-node insert (+0.07%, ranges overlapping). The recommended-subject check (commit 41) reads nothing and runs only when a write puts config fields outside the system graphs, and the merge's system-graph check (commit 44) is one pass over the staged flakes of a merge.

Quiet box. EC2 c7i.4xlarge, the fat-LTO bench profile with shacl, base and head in interleaved rounds; a change counts as a win or a loss only when the base and head ranges don't overlap and the median delta exceeds the bench's budget (5% at small). No loss by the rule. jsonld_parse wins on update_sugar (−14.7%) and flat_envelope (−15.3%; the base there was bimodal, 148 or 170 ms, but the head was at or below it in every round), and annotations was +4.4%, slower in every round but under the budget (before commits 33 and 34; see Annotation parse). insert_formats (JSON-LD and Turtle) is stable. transact_commit was inconclusive on populated_ledger in the first session (+23.9%, with the ranges overlapping) and is stable over the second session's 7 rounds (Commit cost with shacl, in the performance comment).

Before ready

  • Benchmark on a quiet machine (EC2, with shacl): insert_formats, jsonld_parse and transact_commit, no loss by the rule, and transact_commit counted in process (see Performance). That run predates the latest changes.
  • Re-run criterion jsonld_parse (annotations) and insert_formats on the quiet machine at the final head, to confirm Annotation parse's shared-machine numbers.

Follow-ups

Grouped under one tracking issue, #2095. Several are one issue each, so the lines go by issue:

Deviations from the design

The main departures: the config is read once, after staging, so the inline-shapes policy-scope refusal comes after staging; one more cross-ledger brick is fixed outside the listed sites (build_transact_policy_context); participation is per written graph; a governed fast-forward is validated in place rather than through the general merge; and uniqueness comes to branch operations with the shared check tail. Each of them, and the smaller ones, is written out in the deviations comment below.

Commits

62 commits, reviewable in order: 59 on c5874d586, the merge of main at 58b53238a (60), one after it (61), and the merge of main at ba4a982ec (62); every fix commit carries its tests. Commits 17–29 answer earlier review passes, and commits 30–47 answer bplatz's review and a review of those changes. Commits 48–59 answer his review of the previous push (fae15f997), and commit 61 follows the merge of #2052. Against main at ba4a982ec, the diff is 111 files, +19.2k / −2.9k.

The commit list, and the diff by kind
  1. bench: jsonld_parse, a parse-only JSON-LD transaction bench (baseline; no production change).
  2. json-ld: one @graph classifier shared by every reader (graph_shape.rs, is_graph_key).
  3. transact: one lexical graph scope for JSON-LD templates (GraphScope, WriteGraphs, one resolver over dataset_ref::GraphSel).
  4. tests: JSON-LD graph scope end to end, with SPARQL twins.
  5. transact: JSON-LD 1.1 named-graph objects; only envelopes carry txn-meta.
  6. transact: one blank-node label sequence per JSON-LD document.
  7. json-ld/transact: @reverse, @included and @nest; refuse other keywords.
  8. transact: annotation lowering shares the scope; one f:reifiesGraph emitter.
  9. query: node-level @graph in a JSON-LD where is GRAPH sugar.
  10. import: bulk JSON-LD import refuses @graph scoping.
  11. api: config reader normalization + diagnostics.
  12. tests: SHACL tests enable SHACL explicitly (no behavior change; see the test-helper row of the mutation table in the tests comment).
  13. api: explicit-only SHACL posture.
  14. api: resolve validation artifacts only for participating graphs.
  15. api: staged config guard, merged with the reasoning-mode check.
  16. docs: graph scoping, explicit-only SHACL, config checks and recipes.
  17. json-ld: a top-level single bare {"@id"} in @graph is envelope content.
  18. query/transact: a where node's @graph names its graph as templates do (parse/graph_name.rs, shared by the template resolver and the where sugar).
  19. api: one post-staging check tail for every write lane (check_staged_write, WriteChecks).
  20. merge: validate fast-forwards into a target whose config governs writes.
  21. api: the config guard refuses a group split across transactions.
  22. transact: read opts.shapes and opts.uniqueProperties from the body.
  23. api/transact: review nits (annotations under a variable graph, the graph-insert selector message, two diagnostics).
  24. docs: compatibility notes for the review changes.
  25. merge/rebase/revert: re-check the target's config groups.
  26. api: the config guard reads config only for config writes (is_config_field).
  27. query/transact: a where's "default" is the ledger's default graph.
  28. transact: flake generation stores every IRI under its canonical namespace code.
  29. transact: an update's where reads top-level values in the ledger's encoding.
  30. api: governance is read once, from the state before the write (multi-operation requests and transactions governed by the state before them and guarded on their result; the fast-forward config test narrowed to config settings; RequestGovernance, ConfigRead; feat(api): multi-write transactions #2023's Transaction through SequentialStager::stage_with).
  31. api/server: inline uniqueness follows the inline-shapes policy rule.
  32. api: one config vocabulary for the reader, the guard, diagnostics and the recipe (config_vocab.rs; the root subject's fields; every stranded node diagnosed).
  33. json-ld: expansion borrows its path and its context.
  34. transact: annotation nodes are included in their edge's subject (placement by construction; the edge graph on delete; selector graph names; reverse properties).
  35. merge/rebase/revert: refuse carrying another branch's config graph.
  36. api/cli: typed config errors and statuses; tracked info prints diagnostics (409 for branch operations; 500 err:storage/ReadFailure for a config read failure).
  37. push: graph ids come from the ledger's graph registry.
  38. api: a ledger warns at load when it holds shapes no config enforces.
  39. docs: enabling SHACL leads with INSERT DATA; compatibility notes (and the persisted-data notes).
  40. tests: overflow values read from the default graph under a named base graph (the 6a03e83a4 seam).
  41. api: the recommended config subject's fields belong in the config graph.
  42. api: a plain Turtle insert honors inline uniqueness constraints.
  43. api: a commit that declares a property unique makes earlier stages stage again.
  44. merge: a merge does not change the target's system graphs.
  45. docs: fast-forward triggers, branch-operation statuses, latest compatibility notes.
  46. merge: a fast-forward's config-settings test keeps only what a refusal needs (the f: type arm, which no refusal needs, is dropped).
  47. json-ld: a scoped context's term IRIs expand against the active context (predates the PR; found by a differential run of the expander).
  48. query: a CONSTRUCT template node's own @graph names its output graph.
  49. query: one @graph selector path for where nodes, top-level and nested (no behavior change).
  50. api: untyped shapes count as shapes; SHACL switched off is not reported.
  51. api: a plain Turtle insert applies inline shapes and a requested mode.
  52. graphql: sh:minCount makes a field non-null only where SHACL is enforced (the schema cache keys on the view's graph).
  53. rebase: a config refusal names the commit the rebase stopped on.
  54. cli/server: config diagnostics without the full ledger info; MCP lists them.
  55. tests: a typed reverse term's annotation; an anonymous split config group.
  56. docs: JSON-LD deviations; graph names through @vocab; plain Turtle options.
  57. tests: every kind of operation a transaction stages is checked against its policy.
  58. docs: a CONSTRUCT template node's @graph; what counts as unenforced shapes.
  59. query: the CONSTRUCT shorthand keeps a where node's graph.
  60. Merge of main at 58b53238a (fix(server): query routes read a request's dataset the way the engine does #2019, fix(index): every xsd:double value is indexable #2044, perf(index): fewer requests for remote leaf reads; whole-leaf fetch by default #2048, perf(index): reuse zstd contexts per thread for column blocks #2049, fix(api): key cross-ledger governance by the model ledger's head commit #2052).
  61. api: a request reads each model ledger at one head.
  62. Merge of main at ba4a982ec (fix(server): one HTTP status table for API errors #2102, perf(indexer): warm-on-write only the leaflets readers had cached, on a bounded pool #2068, fix(indexer): debug index builds accept f:message, f:author and f:reverts in txn-meta #2056, perf: decode only the columns count fast paths read; one cached entry per leaflet #2053, build(deps): move RustSec-flagged dependencies to patched releases #2046, ci(release): pre-releases publish to Homebrew as fluree-beta #2045, v4.2.4).

Diff against ba4a982ec (111 files, +19.2k / −2.9k): Rust under src/ +9.9k / −2.5k (including in-file test modules), integration tests and shacl_tests.rs +8.7k / −0.4k, docs +0.4k / −0.1k, bench +0.2k.

Tests and non-vacuity

New and changed tests cover every area this touches: JSON-LD scope and named graphs, txn-meta, annotations, the where graph names, the bulk-import refusal, the config reader and diagnostics, the explicit-only matrix, fast-forward validation and uniqueness on branch operations, inline shapes from options and from the body, the cross-ledger fail-closed pairs, and the config guard on every lane. Every fix was checked by reverting only the fix, watching its tests fail, and restoring. The area list and the mutation table are in the tests comment below.

Gates

Rebased onto main at c5874d586 (#2006, #2008, #2013, #2036 and #1451 merged since the previous round; how each meets this PR is in Sequencing above).

At commit 58 (cca4934bd), on a shared Mac under load from other builds, these pass:

  • fmt;
  • clippy -D warnings, --all-features and default features, on api, transact, query, CLI, server, core and JSON-LD, except core's default-feature run (below); both wasm32 clippy runs from CI;
  • cargo check --workspace --all-features --all-targets, and CI's server check without default features;
  • doc tests for JSON-LD, api and query (17 passed, 101 ignored).

nextest:

  • 3,337 passed / 1 skipped over transact, JSON-LD, query and core;
  • 4,486 passed / 21 skipped for fluree-db-api with shacl,graphql;
  • 843 passed / 1 skipped for the api lib without SHACL;
  • 769 passed over consensus and server;
  • 470 passed for the CLI;
  • 1,286 passed over docs, memory, MCP, Cypher, SHACL, policy, reasoner, SPARQL, nameservice-sync and GraphQL.

Commit 59 (7125b79dc) changes the query parser's CONSTRUCT shorthand, a test and two doc lines; for it, fmt, clippy -D warnings on query and api (all features and default), 1,723 query tests, 97 CONSTRUCT and graph-scope tests across the api test binaries and the query doc tests pass, and its test fails with the fix reverted.

Commit 61's test (0454f401e) fails with the request's head pins removed, and with a rollback's restage dropping them.

At the head (cf1d8b586, after the merge of main at ba4a982ec): fmt; cargo check --workspace --all-targets, default and all features; clippy -D warnings on api, transact, consensus, server and SHACL, all features and default; nextest 4,531 passed / 21 skipped for fluree-db-api with shacl,graphql (it_transaction, it_policy_cross_ledger and it_shapes_cross_ledger included), 850 for the api lib without SHACL, 3,397 over transact, JSON-LD, query, core and policy, and 779 over consensus and server. The status-table test fails with a config read failure classified 400.

Not run locally, or not clean:

  • The default-feature clippy of fluree-db-core alone fails on an unused test import in commit.rs, which this PR does not touch and which fails the same way on main. The workspace-level clippy CI runs unifies features and does not see it.
  • The workspace-wide --all-features nextest that CI runs (api's iceberg, sql, delta, aws, credential and vector targets, and the crates not listed).
  • The Python package's pytest suite (CI builds the wheel and runs it).
  • testsuite-shacl and testsuite-sparql; this PR touches neither.
  • The wasm probe and browser jobs; the live SQL bridge lane.
  • The criterion benchmarks at this head. Performance lists what was measured (Before ready).

@bplatz

bplatz commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

This also fixes #1991, which the description doesn't reference. All three repro cases from the issue fail on main (c5874d586) and pass on this branch at fae15f997:

  • ["graph", g, …] insert with @annotation: the annotation lands in g with its edge. On main it goes to the default graph and attaches to a default-graph edge with the same subject, predicate and object.
  • The same shorthand in a delete: retracts the annotation in g. On main the annotation stays attached.
  • Top-level "graph" key with an annotated edge: commits, with the annotation in that graph. Main refuses it with GraphMismatch.

Fixes #1991

Could you also put Fixes #1991 in the PR body? A closing keyword in a comment doesn't link the issue or close it on merge.

@bplatz

bplatz commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Filed #2069: a branch created from a ledger that has no index yet reads none of its source's ledger config. On replay, the source's config lands in a user graph, and the branch's config graph is empty.

This PR didn't cause it, but it widens it.

  • On main today: such a branch already loses configured policy (data the policy hides on main is visible on the branch) and uniqueness. SHACL settings are ignored there too, but the shapes-exist fallback still enforces SHACL.
  • With this PR: SHACL is enforced only where config enables it, so it is off entirely on those branches. fast_forward_into_a_shacl_enabled_target_is_validated already relies on this: it expects dev to take a write that main's config rejects (expect("dev takes the write")).

Could you list it with the breaking change and add Follow-up: #2069 to the description? The alternative is landing the #2069 fix before this.

@bplatz bplatz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of fae15f997 (with main merged). Where a comment says "reproduced", I ran it against both main and this branch. #1991 and #2069 are in separate comments above.

Items without a line anchor:

  • Upgrade visibility:
    • Nothing warns at write time or ledger load when a ledger holds shapes and SHACL is off.
    • MCP's ledger-info output (fluree-db-server/src/mcp/format.rs) doesn't include configDiagnostics, and server-routed fluree info prints the raw JSON.
    • fluree model entity define states the rule but doesn't check this ledger's posture.
  • GraphQL: sh:minCount ≥ 1 still makes a field non-null (fluree-db-graphql/src/schema/build.rs:430). On a ledger without SHACL config nothing enforces it, so a record written without the field reads back as a non-null error. docs/cli/graphql.md and docs/query/graphql.md still say SHACL applies unchanged.
  • Missing from the behavior-change list:
    • @reverse context terms now write the opposite direction. That's correct per spec, but data written by older versions stays inverted, so a delete through the same term no longer matches it.
    • Graph names no longer resolve through @vocab; they're refused as relative.
    • opts.shapes sent with an identity is now a 400.
  • Duplicated predicates:
    • Three definitions of a config field: is_config_field, any f: predicate in graphs_with_config_fields, and carries_config_settings.
    • A graph IRI that can't be resolved is handled two ways: stored_shacl_policy skips it, while inline_shapes_policy and resolve_per_graph_unique_sids fall back to the ledger-wide config.
    • The @graph selector block is duplicated in parse_node_map and parse_nested_node_map.
  • Coordination: #2007 adds its own fluree-db-core/src/dataset_ref.rs with a different GraphIri / RefError. Whichever lands second has to reconcile them.
  • Description:
    • #1979 was closed by #2022 (items 1 and 3), so the #1979 framing is stale.
    • The follow-ups marked "not filed yet" should be filed and linked as Follow-up: #N.

Comment thread fluree-db-query/src/parse/node_map.rs Outdated
// the same key does in an insert or delete (`resolve_where_graph`), so
// one document means one graph by it. (It used to be read as a predicate
// named `@graph`, so the pattern matched nothing.)
if let Some((graph_key, raw)) = node_graph_selector(map, context)? {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A CONSTRUCT template node with @graph now becomes a Graph pattern here, and parse_construct_items (parse/mod.rs, the filter that keeps only Triple / EdgeAnnotation) drops it. Reproduced: "construct": [{"@id": "?s", "@graph": "ex:g2", "ex:label": "?n"}] returns "@graph": [] with no error, while main returns the triples. Nothing tests a CONSTRUCT template with @graph.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, a real regression. A template node's own @graph is now GRAPH sugar in a template too: parse_construct_items keeps the Graph pattern (and the triples and annotations inside it) instead of filtering it out, so your "construct": [{"@id": "?s", "@graph": "ex:g2", "ex:label": "?n"}] writes its triples into ex:g2, exactly as ["graph", "ex:g2", {…}] and SPARQL's CONSTRUCT { GRAPH ex:g2 { … } } do (77592cadd). Test: jsonld_node_level_graph_in_a_template_names_the_output_graph compares both JSON-LD forms with the SPARQL one as N-Quads. The CONSTRUCT page shows the node form, and the compatibility notes list the change from main, where @graph was read as a property and the triples went to the default graph (cca4934bd). The "construct": true shorthand had the same filter, so a graph-scoped where node matched and wrote nothing there too; it now goes through the same function (7125b79dc, jsonld_construct_shorthand_keeps_a_where_nodes_graph).

Comment thread fluree-db-query/src/parse/mod.rs Outdated
// A node-level `@graph` in `where` can name a `fromNamed` alias, which
// must stay the alias rather than expand as an IRI.
let ctx = JsonLdParseCtx::new(context.clone(), path_aliases, parse_policy).with_graph_names(
GraphNameEnv {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A ["query", …] subquery in a where is parsed through here with a fresh GraphNameEnv (no ledger, no outer aliases, no ledger_default_graph), so it resolves graph names differently from the enclosing where. Reproduced: in an update with "graph": "ex:g", {"@id": "?s", "@graph": "default", "ex:p": "?o"} reads the default graph at the top level but ex:g inside ["query", {…}]. Passing the parent's environment through would fix it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was fixed after the head you reviewed: a ["query", …] subquery takes the naming environment of the where it sits in, the ledger, the fromNamed aliases and the ledger default graph included (2d5824671). subquery_names_graphs_as_its_where_does is your repro: an update with "graph": "ex:g" whose subquery reads {"@id": "?s", "@graph": "default", "ex:p": "?o"} moves the default graph's data into ex:g (with and without a restricting fromNamed), and the subquery sees the update's aliases.

// A reverse property's edge points at this node, and a nested
// value's properties are this node's: an annotation inside either
// would be lowered against the wrong edge.
Some(keyword @ ("@reverse" | "@nest")) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refuses annotations under the @reverse keyword only. A context term defined with "@reverse" falls through. Reproduced with {"parent": {"@reverse": "ex:child", "@type": "@id"}} and {"@id": "ex:kid", "parent": {"@id": "ex:mom", "@annotation": {"ex:since": 2020}}}: the edge is stored as ex:mom ex:child ex:kid (now correct), but the annotation reifies ex:kid ex:child ex:mom, which doesn't exist. It commits without error in the default graph. On main both used the inverted direction, so they matched.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also fixed after that head (4c5e4c050): one edge-key classifier, shared by the insert and delete walkers, tells a keyword, a reverse-defined term and a forward term apart, so an annotation on a value of a reverse term reifies the edge as stored (ex:mom ex:child ex:kid), and the same document deletes it. Your exact context ({"parent": {"@reverse": "ex:child", "@type": "@id"}}) is now a case in annotation_on_a_reverse_property_reifies_the_stored_edge, next to a term without @type and the @reverse keyword.

/// every graph (ledger-wide and each graph override) and the shapes source
/// holds at least one `sh:NodeShape` or `sh:PropertyShape`. A config that
/// enables SHACL for one graph only is configured.
async fn shapes_without_enforcement(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This counts only subjects typed sh:NodeShape / sh:PropertyShape, but the compiler also compiles untyped shapes, which the old rule enforced. Reproduced: ex:PersonShape sh:targetClass ex:Person ; sh:property [sh:path ex:name ; sh:minCount 1] with no config enforces on main. Here it stops enforcing and configDiagnostics is empty.

Separately, it also reports shacl-not-configured when the config explicitly sets f:shaclEnabled false.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Untyped shapes not being enforced without config is the explicit-only rule itself (the breaking change), but the diagnostic and the load-time warning should still have flagged them, and now do: a subject with a SHACL target predicate (sh:targetClass, sh:targetNode, sh:targetSubjectsOf, sh:targetObjectsOf) counts as a shape, typed or not. The load gate reads the index's property stats and the novelty for those predicates, so it still doesn't read the ledger for a ledger without shapes. And shacl-not-configured is no longer reported when the config sets f:shaclEnabled false (ledger-wide or for a graph): that's a posture, not a gap. (f25bfb626) Tests: diagnostics_report_shapes_without_enforcement (an untyped shape reported; switched off not reported) and the load-warning binary (untyped, in novelty and indexed; switched off silent). fluree info's page and the compatibility notes say what counts (cca4934bd).

Comment thread fluree-db-query/src/parse/graph_name.rs Outdated
/// `"default"` writes. It contains a space, which no IRI can, so no graph
/// written under an IRI has it; an update refuses it as a graph name or VALUES
/// value it writes ([`reserved_graph_name_refusal`]).
pub const LEDGER_DEFAULT_GRAPH: &str = "@ledger default";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: #2005 already makes urn:default name the ledger's default graph in an update's WHERE (stage.rs registers it as a non-enumerable alias for g_id 0). Resolving "default" in an update where to urn:default would remove the need for this constant, its refusals (VALUES, fromNamed, node-level and ["graph", …]), the registered-name collision error, and the Turtle \u follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and done after the head you reviewed: a where node's "@graph": "default" resolves to urn:default, #2005's name, and the sentinel, its refusals and the registered-name collision are gone (2d5824671).

Comment thread fluree-db-api/src/tx.rs
&mut view,
&mut ns_registry,
&rustc_hash::FxHashMap::default(),
WriteChecks {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plain-Turtle branch passes only tracker here. The txn_opts that stage_turtle_insert_with_opts receives (inline shapes, unique properties, validation mode) reach the checks only on the TriG fallback. So insert_turtle(..).txn_opts(..) with inline shapes commits unchecked. That predates this PR, but it contradicts "read on every surface".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: the plain Turtle lane now passes the caller's inline shapes, requested validation mode and inline uniqueness constraints (and the policy context, so policy-scoped inline constraints are refused as elsewhere) to the shared checks, as TriG does (fdaa4fdd5, 5d27ae6a9). Tests: inline_unique_property_applies_to_turtle_with_and_without_graph_blocks and inline_shapes_apply_to_turtle_with_and_without_graph_blocks (each refuses in reject mode and commits in warn mode, plain and TriG).

/// Where inline shapes are permitted, a requested warn mode applies to them:
/// the violation is logged and the write commits.
#[tokio::test]
async fn inline_shapes_honor_a_requested_warn_mode() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only asserts that the write commits, which also holds with the inline pass removed. Asserting the warning, or that the same request fails in reject mode, would pin it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: inline_shapes_honor_a_requested_warn_mode now also sends the same write in the default mode and asserts the SHACL violation, so it fails with the inline pass removed (5d27ae6a9).

Comment thread fluree-db-api/tests/it_config_graph.rs Outdated
/// returns.
#[cfg(feature = "shacl")]
#[tokio::test]
async fn repair_recipe_restores_a_split_config() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The split group is seeded as the IRI urn:config:shacl. A ledger split the documented way (#1979) has an anonymous group, so a blank node. One case with a blank-node group would cover the ledgers that will actually run this recipe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added: repair_recipe_restores_a_split_config has an anonymous-group case (the group a blank node, its fields in the default graph), and the docs recipe repairs it verbatim (b55c02aa0).

Err(TransactError::ShaclViolation(report).into())
}
Some(rejection) => {
let _ = describe;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For uniqueness and config-guard refusals describe is dropped, so a rebase that stops on one doesn't say which commit. No rebase test covers either.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed for config refusals: a rebase stopped by one now says replaying commit t=N was refused: … (409), like a SHACL stop (f89b0a638). A uniqueness violation is returned as its typed error, which has no free-text field to carry the commit, so it doesn't name the commit yet; that's in #2077. Tests: rebase_refused_by_a_config_check_names_the_commit and rebase_rejected_when_replay_violates_uniqueness.

Comment thread docs/reference/compatibility.md Outdated
- Nested objects
- Arrays

Fluree extension: a node's own `"@graph": "<graph IRI>"` writes the node, and every node nested in it, to that graph (`"default"` and `"config"` name the default graph and the ledger's config graph). `@index` is accepted and ignored. Other `@`-keywords on a node object are refused.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unknown @ keywords are refused here, where the JSON-LD 1.1 expansion algorithm drops them, so "Fully compliant" above should list this as a deviation. Also, the @json literals bullet below now hangs off this paragraph instead of the list.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: the JSON-LD status is now "compliant, with the deviations listed below", and the deviations are listed (unknown @-keywords refused, plus two scoped-context differences a corpus run found, which are follow-ups). The @json bullet is back in the list (68bc66e33).

@bplatz bplatz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: the shared check step, fast-forward validation and graph scoping are a solid direction, and CI is green with main merged (fae15f997).

Please work through the review comments before merging, in particular:

  • the four reproduced regressions: CONSTRUCT dropping @graph nodes, reverse-term annotations, subquery graph names, and the untyped-shape diagnostic
  • #2069, which this PR widens
  • Fixes #1991 in the description

The rest can be fixed here or filed and linked as follow-ups.

aaj3f added 11 commits October 12, 2026 08:09
Times fluree_db_transact::parse_transaction alone (context handling,
edge-annotation lowering, expansion, template construction) with no
ledger, staging or commit, so parser changes are attributable. Four
shapes: a flat insert envelope, depth-3 node trees under a node-level
graph selector, an update of ["graph", g, node] items, and annotated
edges. Registered in regression-budget.json and the bench categories
table.
graph_shape::classify_graph_value decides what a node's @graph value is:
a graph selector (a string, a bare {"@id"} object, or a one-element
array of either), JSON-LD 1.1 named-graph content (node objects), or
invalid (numbers, booleans, null, value objects, nested arrays).
doc_shape classifies a top-level object as an envelope, a named graph or
a node from the same answer, and is_graph_key recognizes @graph, a
context alias of it, and the legacy bare `graph` key only when the
context does not define `graph` as a term.

The expander adopts both: a context term `graph` is now an ordinary
property at every level instead of being hijacked as @graph (the triple
was dropped and the node re-routed), and a top-level @graph is an
envelope only when it holds node objects and the object has no @id; a
top-level graph selector expands as a node.
A JSON-LD node's statements, and those of every node nested in it, now
land in the node's scope: its own graph selector if it has one, else the
enclosing node's. Before, only a node's own triples followed its
"@graph" selector; nested nodes, anonymous nodes and @list items fell back
to the default graph (or the update's "graph" key), so the nested config
groups of #1979 landed in the default graph, a delete or upsert of a nested
node under a selector retracted in the wrong graph, and list members'
properties split from their list.

- GraphScope (ir.rs) is passed down the recursion by reference; every
  value/list/node function takes one, so a nested node cannot be emitted
  without a scope. WriteGraphs::scope is the only way to get a named
  scope, so every graph a template names is registered as a write target.
  The ambient TemplateParseCtx::default_graph and its save/restore are
  gone.
- One graph-name resolver for node selectors, ["graph", g, ...] items and
  the update "graph" key: a variable (update templates only) is the WHERE
  binding (TemplateGraph::Var, as SPARQL GRAPH ?g), a fromNamed alias its
  IRI, then the keyword table (dataset_ref::GraphSel, new in
  fluree-db-core): "default" is the default graph everywhere, "config" the
  ledger's config graph, "txn-meta" is refused; anything else expands
  @id-style and must be an absolute IRI. A variable graph used to write a
  graph literally named "?g"; "default" meant a graph named "default"
  outside the update key; relative names minted relative-IRI graphs.
- Graph insert / sync parse with the request's graph as the fixed root
  scope instead of re-homing templates afterwards; a selector naming any
  other graph is refused.
- parse_transaction / parse_graph_insert / parse_sync_transaction take the
  target ledger id (for the "config" keyword).
The #1979 recipe from docs/ledger-config/writing-config.md, verbatim:
all four statements land in #config and the configured SHACL refuses
the forbidden record. Nested @id, anonymous and @list-item nodes land in
the selector's graph (P1, P10'); a node selector beats the update graph
key for its subtree (P7c); delete and upsert of a nested node act in its
graph (P7e, P9); a variable graph writes the WHERE-bound graph; the
"config" keyword writes #config. Each compares against the SPARQL
spelling of the same write where one exists.

Renames it_config_graph::config_write_json_ld to
config_write_trig_iri_groups: its body writes TriG.
A node object with @graph content, {"@id": G, "@graph": [nodes]}, is a
JSON-LD 1.1 named graph (section 4.9): the content nodes are written to G
and the node's own properties stay in the enclosing scope. Named graphs
nest (innermost wins) and may appear as property values. Before, the
content was dropped and the first content node's @id was read as the
node's graph selector; a property-bearing object @graph lost its data the
same way.

- A named graph needs an IRI name: a blank-node or variable name, or a
  graph object without @id below the top level, is refused.
- Graph insert / sync refuse named-graph objects (the request names the
  graph). {"@id": G, "@graph": []} is no longer mistaken for the
  explicitly empty envelope a sync may clear a graph with.
- txn-meta extraction and the reserved-key strip use doc_shape: only an
  envelope's top-level keys are metadata. A single node carrying a graph
  selector (P10e) and a named graph's own keys are data; they used to be
  written as data and also as txn-meta on the commit.
- doc_shape and the expander: at the top level, an array @graph without
  @id is an envelope even when it holds one bare {"@id"} node, as every
  reader has always treated it; a single property-bearing object @graph
  is an envelope too (the single-node write form some clients send).
Anonymous nodes are labelled _:b0, _:b1, ... by one issuer for the whole
transaction body. The counter used to restart for every expansion call,
so the anonymous nodes of different ["graph", g, ...] items (and of the
plain items after them) all became _:b0 and merged into one node.

A user-written label of the issuer's own form (_:b3) would merge with the
anonymous node given that number. The parse notes such labels (templates
and VALUES), and when one was also issued it parses the document once
more with the issuer skipping those numbers. The positional labels are
part of a stored blank node's identity under upsert and sync, so their
spelling is unchanged; only documents that were being corrupted get
different labels.

The edge-annotation firewall now refuses user labels in the lowering's
reserved _:fluree_ann_ prefix, which would merge with the lowering's
anonymous annotation nodes.
Keywords on node objects no longer leak into the data as predicates named
after the keyword:

- @reverse (JSON-LD 1.1 section 4.8): each value is the subject of a
  statement pointing at the node, written in the node's scope. The
  expander used to write a context term defined with @reverse as a
  forward property, inverting the statement; it now expands into the
  node's @reverse map, as the keyword does.
- @included (section 4.7): the included nodes are written in the node's
  scope with no linking statement.
- @nest (section 4.6): the expander merges a nested value's properties
  and types into the enclosing node, recursively; a nested value cannot
  carry @id or @graph.
- @index carries no statement and is dropped.
- Any other key of JSON-LD keyword form (@ then letters, e.g. @language
  on a node) is refused. Keys that merely start with @ (@odata.etag) are
  ordinary property names, as before.

Bulk import reads @reverse and @included the same way (they were
dropped). The edge-annotation lowering treats @included nodes as nodes
and refuses an annotation inside @reverse or @nest, which it would have
lowered against the wrong edge.
…itter

The edge-annotation lowering computed "which graph is this edge in" on
its own, and disagreed with the parser: it ignored the update's graph key
(A1: the annotation was refused as a GraphMismatch), the ["graph", g, ...]
item (A2) and the array selector (A3), writing the bundle to the default
graph without an anchor while the edge was in g, and never lowered
annotations inside named-graph content.

- The walkers read @graph with the shared classifier and is_graph_key
  (aliases included; the root document's bare `graph` routing key is not a
  selector), recognize ["graph", g, ...] items in update clauses, and
  descend into named-graph content with the graph's @id as the scope.
  Siblings are placed with the edge's scope as their graph selector, which
  the parser resolves exactly as it resolved the edge's. attach_siblings
  reads the document shape: a top-level named graph is re-wrapped, not
  appended into its content.
- GraphScope::emit is the one template-level writer of f:reifiesGraph: a
  reifier's f:reifiesSubject emitted in a named scope brings
  (reifier, f:reifiesGraph, scope graph) with it, for assertions and
  retractions alike. The walkers' stamps and graph insert's post-hoc
  anchor loop are deleted.
- With #1997 and #2005, an update's `graph` key that names the default
  graph (the ledger's own address, or `urn:default`) writes the default
  graph: staging moves the templates of that template default there. An
  annotated edge under it now moves with its bundle, and staging drops
  the bundle's anchor, which named that graph: a default-graph bundle has
  none, so the annotation is found with its edge in the default graph.
- A fail-closed cross-check: after an annotated document parses, every
  reifier bundle must reify an edge asserted in the bundle's own graph.
  It runs once any template names a graph (a document entirely in the
  default graph has no scoping to disagree about) and allocates nothing
  per template, which keeps the annotation parse bench within noise.
A node-level "@graph" selector in a where node-map now scopes that
node's patterns (and those of nodes nested in it) to the named graph,
exactly as ["graph", <name>, {...}] does, so the same key means the same
scope in where, delete and insert. It used to be parsed as a predicate
named @graph: a query matched nothing, and an update's WHERE matched
nothing, so the update committed zero flakes and its target survived.

The name follows the existing where-clause GRAPH rule (an alias or the
graph's IRI as written); a context alias of @graph works the same way,
and a bare `graph` key stays an ordinary property in a pattern. A node
with @graph content (a JSON-LD named graph) is refused in a where.
The bulk-import JSON-LD adapter skipped every @-key but @type, so a
node's graph selector was dropped (its statements folded into the
default graph) and a JSON-LD named graph's content never reached the
sink. The GraphSink contract calls that folding data loss. The adapter
now refuses a node-level @graph (selector or content) with a message
pointing at `fluree insert` or a TriG / N-Quads import. Writing graph
scopes from bulk JSON-LD (ImportSink quads) is left for a follow-up; a
top-level envelope is unwrapped by expansion and imports as before.
The config reader took the first binding of a setting-group pointer, so
a config with two f:shaclDefaults values (a re-inserted config) could
read either group, and a group whose fields were written outside the
config graph (the #1979 shape) read as a present-but-empty group.

- A group that sets nothing in the config graph reads as absent. That is
  observably the same as before for every group; it makes the reader
  deterministic and keeps "empty group" from diverging from "no group".
- Several values of one pointer: empty groups are skipped and the rest
  ordered by decoded IRI, the rule several f:LedgerConfig subjects
  already follow. A rate-limited warning names the pointer.
- config_resolver::diagnose reports degenerate config state: several
  f:LedgerConfig subjects, ambiguous group pointers, empty groups and
  the graphs their fields were stranded in, and shapes that no config
  enables ("shapes present; SHACL enforcement not configured").
- Ledger info carries them as `configDiagnostics` (omitted when there
  are none), and `fluree info` prints them.
aaj3f added 9 commits October 12, 2026 10:49
A Turtle insert's programmatic options reached the checks only when the
document had a `GRAPH` block, which sends it through the TriG path: a
plain Turtle document ignored `TxnOpts::shapes` and
`TxnOpts::validation_mode` and committed unchecked. The plain lane now
passes them to the shared checks with the inline uniqueness
constraints, as the TriG path does; a build without SHACL support
refuses inline shapes there too, as on the other lanes.

Tests: inline_shapes_apply_to_turtle_with_and_without_graph_blocks
(plain and TriG, refused in reject mode, committed in warn mode).
inline_shapes_honor_a_requested_warn_mode now also asserts that the
same write is refused in the default mode, so it fails with the inline
pass removed.
`sh:minCount >= 1` made a GraphQL field non-null. With SHACL enforced
only where the ledger config enables it, nothing keeps a record on
another ledger from lacking the value, and a non-null field turns such
a record into an error on read. The schema now marks a field non-null
from `sh:minCount` only where the config enables SHACL for the view's
graph (the view's resolved config, or the ledger's own when the view
was built without it); elsewhere the field is nullable. The derived
schema's cache keys on the view's graph, since the schema now depends
on it. The CLI and query GraphQL pages say so, and that SHACL
validation of mutations applies where SHACL is enabled.

Tests: min_count_makes_a_field_non_null_only_where_shacl_is_enforced,
min_count_follows_the_views_graph (a view of a graph with SHACL
enabled, derived after a view of the default graph at the same t). The
other shape tests seed a SHACL-enabled ledger.
A rebase replays the branch's commits one at a time and names the
commit it stopped on, but only for a SHACL violation: a refusal from
the config checks (a split group, a second value) came back without
it. Branch-operation refusals from the config checks now go through
the same description, `replaying commit t=N was refused: ...`, still a
409. A uniqueness violation is still returned as its typed error, which
carries no free text for the commit.

Tests: rebase_refused_by_a_config_check_names_the_commit,
rebase_rejected_when_replay_violates_uniqueness.
…s them

Local `fluree info` built the full ledger-info statistics only to read
`configDiagnostics`. It now calls `config_resolver::diagnose` on the
loaded ledger. The MCP server's ledger-info markdown gains a
"Config Diagnostics" section when there are any, so an agent reading
the data model sees, for example, shapes that no config enforces.

Test: test_format_data_model_config_diagnostics.
…roup

annotation_on_a_reverse_property_reifies_the_stored_edge gains a
reverse term with `"@type": "@id"`. repair_recipe_restores_a_split_config
gains a group that is a blank node with its fields in the default
graph, as the documented JSON-LD form wrote one before this branch; the
docs recipe repairs it verbatim.
…tions

The JSON-LD 1.1 status lists its deviations instead of "fully
compliant": unknown `@`-keywords on a node are refused where the
expansion algorithm drops them, and two scoped-context differences (a
type-scoped context propagates into nested nodes, and a term defined
only in a property-scoped context stays a bare key outside it). The
`@json` bullet is back in the feature list. The behavior notes add that
a graph name no longer expands through `@vocab`, and that a plain
Turtle insert applies inline shapes and a requested validation mode.
…t its policy

A transaction stages JSON-LD, Turtle and SPARQL operations through different staging lanes. Under a deny-all policy each kind is refused, and the ledger does not move.
…hapes

The CONSTRUCT page shows a template node's own `@graph` beside the
`["graph", …]` form. The compatibility notes list it, the GraphQL
nullability rule and what the `shacl-not-configured` diagnostic counts
(typed shapes and subjects with a SHACL target, not a config that sets
`f:shaclEnabled false`), which `fluree info`'s page now says too. Two
GraphQL paragraphs are rewrapped.
`"construct": true` uses the `where` clause as the template, but it
kept only the clause's top-level triples and edge annotations, so a
`where` node with its own `@graph` (a graph block) matched and wrote
nothing. The shorthand now takes its template through the same
`construct_template_pattern` as an explicit template, so the node's
triples come out in its graph. SPARQL's `CONSTRUCT WHERE` has no
`GRAPH` form, so it is unchanged.

Test: jsonld_construct_shorthand_keeps_a_where_nodes_graph (a default
graph node and a graph-scoped node, against the explicit SPARQL
template, as N-Quads).
@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Performance details

The measurements behind the PR body's Performance section.

Wall time, median over 6 rounds (load 9-17). Push and loads are on file storage; each push goes into a fresh copy of the base:

Case main before this head vs main vs before
push, 20 commits, no config 16.0 ms 16.1 ms 16.0 ms +0.3% -0.4%
push, 20 commits, governed 16.7 ms 16.7 ms 16.6 ms -0.7% -1.0%
cold load, indexed, no shapes 371 µs 342 µs 369 µs -0.5% +8.1%
cold load, indexed, shapes and no config 342 µs 361 µs 437 µs +27.8% +21.0%
cold load, unindexed, shapes and no config 7621 µs 7654 µs 7942 µs +4.2% +3.8%
fast-forward, ungoverned, unindexed, 1000 commits, fresh 16.3 ms 166.6 ms 166.7 ms +925.4% +0.1%
fast-forward, ungoverned, unindexed, 1000 commits, cached 18.1 ms 17.8 ms 18.5 ms +2.1% +4.0%
fast-forward, ungoverned, indexed, 1000 commits, fresh 16.9 ms 19.0 ms 20.6 ms +21.7% +8.8%
fast-forward, SHACL enabled, unindexed, 1000 commits, fresh 15.8 ms 174.9 ms 172.4 ms +994.3% -1.4%
  • Push is unchanged: graph ids come from the registry instead of the binary store, one lookup per distinct graph.
  • The load-time warning costs one index lookup, on a ledger whose index stats show shapes and whose config does not enforce them. That is the +76 to +95 µs on an indexed ledger with shapes. A ledger without shapes pays two in-memory checks; its row is noise. A ledger whose config enables SHACL reads its config at load, and that read is put in the handle's config cache, so the first request skips it.
  • Fast-forward: this head matches the branch before these changes in every cell. These sources carry no annotations, so the rows do not exercise the narrowing of carries_config_settings to config settings; a separate run whose sources carry edge annotations (9 rotated rounds) put its saving at 0 to 1.3 ms where staging would otherwise run (cached and indexed targets), and none on a fresh unindexed target, whose cost is the target load. The fresh, unindexed cost is the target load needed to learn whether the target governs writes. It is unchanged, as decided, and the parity design is a next follow-up (Persisted config summary, so a fast-forward doesn't load an unindexed target (next) #2093). Fast-forwards into a governed target are validated (commit 20), which main does not do.

Commits 28 and 29. An in-process harness (not committed) counts instructions retired and allocator calls per transaction (dev-fast, default features, Apple M4; the lane table above is with shacl). It runs 40 transactions after a 5-transaction warm-up on a memory ledger of 1,000 subjects: 8 processes per build, rotated, median. Commit 27 → 28 → 29, and main (6b9d5b619) against this head:

Scenario 28 vs 27 29 vs 28 this head vs main
insert, 100 new nodes +0.38% -0.13% +0.69%, allocations 15,704 → 15,306
upsert, 100 nodes +0.03% -0.01% +0.31%, allocations 16,626 → 16,428
update, where binding 100 subjects, insert writing them +0.05% +0.02% +0.14%, allocations 1,959 → 1,961
update, top-level values IRI in a new namespace each time, STR stored +0.03% +0.47%, allocations 7,025 → 6,829 +0.57%, allocations 7,022 → 6,829
the same IRI written as a ref object into 100 flakes +0.03% +0.11%, allocations 3,663 → 3,668 +0.14%, allocations 3,660 → 3,668

The last row was +0.97% (allocations 3,663 → 3,766) before flake generation cached each re-encoded IRI per transaction; it is now +0.11%. main measures a different answer on the STR row (it stores the encoded form).

The canonical write is one comparison per written Sid: commit 28 costs +0.38% on the 100-node insert (its ranges do not overlap commit 27's) and at most +0.05% elsewhere. Commit 29's cost is in the last two rows, which run only when an update binds an IRI in a namespace the ledger has not registered. There the where reads it in the ledger's fallback form, and the first flake written with it re-encodes it; a per-transaction cache serves the rest (the harness uses a new namespace per transaction, the worst case). In the STR row the update also stores a different value: the IRI, where it used to store the encoded form. A first design gave the where the transaction's new codes by copying the ledger snapshot per update: +1.44% instructions and +144 allocations on that same STR row, growing with the namespace table. It was replaced before review.

Commit cost with shacl. On EC2 (c7i.4xlarge, the bench profile with shacl, 7 alternating rounds against main at 61b836e9a), transact_commit is stable by the rule: fresh_ledger −9.4% and populated_ledger +1.7% (the head slower in 6 of 7 rounds). Three of the rounds put both builds into a slower mode, in which the head was slower (populated_ledger +9% to +25%, fresh_ledger +5% to +8%); in the other four it was at parity on populated_ledger (−0.4% to +1.7%) and 9-10% faster on fresh_ledger. To see whether the head does more work per commit, a harness (not committed; dev-fast profile, shacl) ran the two scenarios as the bench does (memory storage, a current-thread runtime, one 10-node Turtle commit on a fresh ledger or on one holding 1000 nodes) and counted per commit: instructions and cycles retired (process-wide), allocator calls and bytes (a counting global allocator), page faults and context switches, and storage calls (a counting wrapper of the memory storage). Base 61b836e9a against this head, 4 interleaved rounds of 290 fresh and 50 populated commits, medians:

Per commit fresh: base → head populated: base → head
instructions 1,774,416 → 1,576,952 (−11.1%) 1,799,812 → 1,769,874 (−1.7%)
cycles 393,332 → 364,470 (−7.3%) 509,230 → 505,758 (−0.7%)
allocations 1,150 → 828 (−28%) 874 → 842 (−3.7%)
bytes allocated 384,851 → 287,954 (−25%) 372,465 → 281,306 (−24.5%)
frees 870 → 555 638 → 608
storage calls 1 write (781 bytes); no reads, lists, deletes or syncs, either build 1 write (795 bytes); no reads, lists, deletes or syncs, either build
page faults, context switches at most 1, none, either build none, none, either build
  • The head does less per commit, on both scenarios. The saving is in the SHACL posture: on a ledger without config, base prepared shape validation on every commit (the shapes-exist heuristic), and the head returns before it. Both builds read the config once per commit, and the head's config-guard pass is inside these numbers.
  • The bench's measured routine also drops the ledger and the Fluree handle. That costs about 110,000 instructions and 230 frees on a fresh ledger, and 2.25 million instructions and 9,700 frees on the populated one (more than the commit itself), the same in both builds (−0.1%).
  • The bench uses memory storage, so a commit does no file I/O and no fsync in either build. The slower mode on EC2 is therefore not extra work, allocation or I/O in the head's commit path; a per-iteration perf stat on that box would separate instructions from cycles there.

The fast-forward validation cost in full, since it's the one new cost on a user-visible operation:

Fast-forward merge (commit 20). A timing harness (not committed; file storage, dev-fast profile) fast-forwarded a branch of 3 commits (20 nodes each) into a target of 100 or 1000 commits (10 nodes each), 15 merges per cell per round, this branch against itself without commit 20, before its rebase onto 61b836e9a, the two interleaved over 2-4 rounds (load 9-19). "Cached": the target is in the ledger cache, as on a server that has read it. "Fresh": a new instance, as a CLI run or a server whose cache no longer holds the target. Medians:

Target Cached: before → after Fresh: before → after
ungoverned, indexed, 100 commits 15.2 → 15.7 ms 15.8 → 17.7 ms
ungoverned, indexed, 1000 commits 16.8 → 19.2 ms 17.8 → 23.0 ms
ungoverned, unindexed, 100 commits 15.6 → 15.6 ms 16.9 → 36.5 ms
ungoverned, unindexed, 1000 commits 21.0 → 21.6 ms 20.4 → 150.2 ms
SHACL enabled, indexed, 100 commits 16.2 → 17.2 ms 17.0 → 20.4 ms
SHACL enabled, indexed, 1000 commits 17.5 → 19.6 ms 19.0 → 23.8 ms
SHACL enabled, unindexed, 100 commits 15.8 → 16.0 ms 16.7 → 34.5 ms
SHACL enabled, unindexed, 1000 commits 22.0 → 22.5 ms 22.1 → 165.0 ms
  • With the target cached, the merge reads the config from the cached state: +0 to +0.6 ms on the unindexed targets and +0.5 to +2.5 ms on the indexed ones (whose config read goes to the index), governed or not. Validating the three adopted commits is within that.
  • A fresh instance has to load the target to read its config, which costs what a first query on the target costs: measured alone, about 2 ms for these indexed targets, and a replay of the unindexed novelty otherwise (19.6 ms for 100 commits, 126 ms for 1000). The CLI in local mode runs without an indexer or a ledger cache, so a fluree branch merge into a target with a long unindexed history pays that replay, as every CLI command that reads the target does.
  • Merges are not a hot path: a cached target pays +0 to +2.5 ms, and the replay a fresh CLI instance pays is the same replay any CLI command pays on unindexed history. Scanning only the config graph during that replay goes with Persisted config summary, so a fast-forward doesn't load an unindexed target (next) #2093.
  • Every fast-forward, governed or not, builds the target's pre-merge state (fast_forward_base: the cached state, or a full load when the cache does not hold the target) and reads its config. The ungoverned rows above measure that cost on its own; it is consistent with the merge cost accepted in the previous bullet.
  • Commit 25 adds, for a fast-forward into a target that governs nothing, a read of the adopted commits to see whether they carry config settings: +0.2 to +0.6 ms on these targets (this head with and without commit 25, 100 commits, indexed and unindexed, 3 interleaved rounds). For a governed target, its extra pass over the staged flakes is within noise (-0.4 to +0.9 ms).

The criterion table, the annotation fix, and the per-commit costs of commits 17–27

Criterion, FLUREE_BENCH_SCALE=small, bench profile, default features, the previous base e4793617f against this branch as of commit 16, before its rebase onto bf523e24e. Each bench ran back to back (before, then after) in two rounds on a shared 16-core macOS machine (load 9-22). The untouched Turtle import_bulk path moved between -10% and +18% across the rounds, which is the noise floor here. The "after" build predates commits 17 onward (see below) and the later annotation changes (the cross-check fix and the invalid-@graph scan, then the check's removal with the placement in @included, and the expander change); the current annotation numbers are under Annotation parse. Medians:

Bench Round 1 before → after Round 2 before → after
jsonld_parse flat_envelope (20k nodes) 85.70 → 87.56 ms (+2.2%) 93.14 → 88.45 ms (-5.0%)
jsonld_parse nested_selector (20k nodes, depth 3) 60.28 → 58.65 ms (-2.7%) 61.02 → 59.47 ms (-2.5%)
jsonld_parse update_sugar (1k items) 2.72 → 2.11 ms (-22.4%) 2.74 → 2.14 ms (-22.0%)
insert_formats jsonld 10 txn × 100 nodes 11.74 → 12.45 ms (+6.0%) 12.08 → 12.27 ms (+1.6%)
insert_formats jsonld 10 × 10 1.27 → 1.29 ms (+1.6%) 1.27 → 1.32 ms (+3.4%)
insert_formats trig 10 × 100 14.29 → 14.90 ms (+4.3%) 14.40 → 14.85 ms (+3.1%)
insert_formats trig 10 × 10 1.54 → 1.56 ms (+1.5%) 1.57 → 1.57 ms (-0.3%)
insert_formats turtle 10 × 100 7.74 → 8.01 ms (+3.4%) 7.78 → 7.93 ms (+2.0%)
insert_formats turtle 10 × 10 875 → 873 µs (-0.2%) 876 → 870 µs (-0.7%)
transact_commit fresh ledger 84.3 → 89.5 µs (+6.2%) 86.2 → 86.1 µs (-0.1%)
transact_commit populated ledger 275.1 → 281.1 µs (+2.2%) 266.0 → 239.6 µs (-10.0%)
import_bulk default threads 228.6 → 252.9 ms (+10.6%) 216.9 → 207.1 ms (-4.5%)
import_bulk single thread 222.2 → 262.1 ms (+17.9%) 234.5 → 212.1 ms (-9.6%)

annotation_hydration (8 cases, read path unchanged) swung both ways in both rounds (-6% to +48%, including its non-annotation baseline at +28% / -6%), with no consistent direction.

jsonld_parse annotations (2k annotated edges) first measured +9.9% / +7.8%, over the 5% budget. The cause was the reifier cross-check allocating a key per template; it was made to borrow its keys and run only once a template names a graph (commit 8). Alternating runs, four rounds each: base 19.21 ms, before the fix 21.13 ms (+10.0%), after 19.63 ms (+2.2%). The check was removed later (commit 34); the current numbers are under Annotation parse.

regression-budget.json gets a jsonld_parse entry (10% tiny, 5% small), like insert_formats.

Commits 17-27 on the transaction path, not re-benchmarked with criterion locally (EC2 and in-process counts for transact_commit are under Commit cost with shacl above):

  • Commit 19 moves the post-staging checks into one function; a transaction runs the same checks in the same order. The config read's "writes a user graph?" decision exits early instead of collecting and sorting a graph id per staged flake (the only change of work on the JSON-LD and SPARQL path), and the Turtle insert lane gains the uniqueness check: nothing on a ledger without config, one pass over the staged flakes' graphs on a ledger with one, and the lookups a JSON-LD insert makes where uniqueness is enabled.
  • Commit 21 adds one comparison per staged flake to the guard's existing pass. Its reads run only for a transaction that writes the config graph, or a config field outside the system graphs: the config graph once, plus a new config edge's target in the user graphs. As first written, any f: predicate outside the system graphs triggered them, including every edge annotation's f:reifies*; commit 26 restricts the trigger to the config reader's vocabulary.
  • Annotation writes (commit 26). A timing harness (not committed; memory ledger with a small config, dev-fast profile) ran 60 inserts of 200 annotated edges each and timed the last 50, for base 61b836e9a and this branch before and after commit 26, interleaved over 5 rounds (load 9-10). Over the 250 timed inserts, median (mean) per insert: base 6.90 ms (6.94), before 7.09 ms (+2.7%; 7.04), after 6.96 ms (+0.8%; 6.95). Per-round means: base 6.91-6.96 ms, before 7.01-7.08 ms, after 6.93-6.99 ms.
  • Commit 27 reads "default" under another default graph as GRAPH <urn:default>, a name every update's WHERE dataset already carries (fix(query): 4xx for unknown graphs and commit refs; urn:default names the default graph #2005). It adds one entry, with one pass over the dataset's names, only when fromNamed restricts the named graphs and the where reads "default"; otherwise it adds one pass over the update's fromNamed list (absent in most updates) to check its aliases.

Commit 50 (untyped shapes in the load-time check). A harness (not committed; dev-fast with shacl, Apple M4, data on X10) loads a file ledger of 200 nodes on a fresh instance (ledger_cached), 9 loads per cell per round, commit 49 against commit 50's tree, alternating over 8 rounds (load 6.7-6.9). Median of the per-round medians, and instructions retired per load:

Cell before after wall instructions
no shapes, unindexed 7,931 µs 7,678 µs −3.2% 82.22M → 82.19M (−0.04%)
no shapes, indexed 366 µs 392 µs +6.9% (slower in 4 of 8 rounds) 3.264M → 3.284M (+0.63%)
typed shape, indexed 439 µs 425 µs −3.3% 4.223M → 4.208M (−0.35%)
untyped shape, unindexed 8,698 µs 8,105 µs −6.8% 83.38M → 82.66M (−0.86%)
untyped shape, indexed 374 µs 450 µs +20.4% (slower in 6 of 8) 3.337M → 4.188M (+25.5%)
  • A ledger without shapes pays one pass over the index's property statistics and four novelty range lookups: about 20,000 instructions on the indexed cell, and nothing measurable against an unindexed load. Its wall-time rows are within the run's noise.
  • An indexed ledger whose only shapes are untyped was not looked at before (the check counted only sh:NodeShape / sh:PropertyShape instances), so it now pays the warning's config read and lookup, the same as the typed cell (4.19M against 4.21M instructions). That is the cost the warning already had for typed shapes, now applied where it was missing.

@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Deviations from the design, in full

  • Commit plan. The scope core, nested recursion and variable graph names are one commit (3), as are named-graph objects and the txn-meta envelope rule (5); the end-to-end scope tests are their own commit (4) so the SPARQL twins read on their own.
  • Config read once, after staging. SHACL and uniqueness share one pre-transaction config read, made after staging: the graphs a transaction writes decide whether any enforcement applies, and a config-only write reads none. The inline-shapes policy-scope refusal therefore happens after staging rather than before it (only a refused request pays for that).
  • One more brick, outside the listed sites. build_transact_policy_context resolved a cross-ledger f:schemaSource before its no-policy-inputs shortcut, so on the server and CLI paths an unreachable ontology model ledger refused every write, config repairs included. It now resolves the schema only when it builds a policy context. Policy sources are unchanged and still fail closed (test and mutation included).
  • Participation is per written graph. The default graph participates in stored-shape validation only when the transaction writes it (it used to be included whenever SHACL was enabled ledger-wide, which triggered source resolution for writes that touched no validated graph). What gets validated is unchanged: only graphs with staged flakes ever were.
  • Inline-shapes membership. sh:class membership in the inline pass reads the default graph plus the value-set facts in the request's own bundle (as fluree validate does for inline shapes), not the configured f:shapesSource graph.
  • Policy-scoped requests. Checked at two points: staging (a non-root policy context, or any policy group in the ledger config, ledger-wide or for a graph, other than one that only sets f:defaultAllow true) and the server route (any request policy input: identity, bearer identity, policy class, inline policy, policy values). The refusal is UnsupportedFeature (Unsupported feature: inline request shapes are not supported in a policy-scoped request), a 400 here and in solo's router.
  • Turtle insert. It now resolves a cross-ledger f:schemaSource for SHACL targeting like the other lanes (it passed none), and runs the whole check tail (the config guard, and uniqueness since commit 19).
  • Fast-forward validation in place. A governed fast-forward is validated where it runs rather than routed through the general merge, which would write a merge commit where a fast-forward adopts the source's commits as they are. The source line's net change since the target's head is staged on the target's pre-merge state (from the ledger cache when it holds the target at the merge's head, else loaded) and checked by the shared tail; the commits are then copied as before, and the ref update expects the validated head. Whether a target is governed is a coarse test (SHACL or uniqueness enabled anywhere in its config); the checks then apply each written graph's own settings. Into a target that governs nothing, the fast-forward reads the adopted commits to see whether they carry config settings (a config field, or a write to a config or txn-meta graph of this ledger, the target's or another branch's; an edge annotation's f:reifies* does not count), and stages them for the config-group checks only then. The target's config is read once and passed through the checks.
  • Uniqueness on branch operations. The shared tail brings uniqueness to merge, rebase and revert as well as to the Turtle insert lane an earlier review pass named: keeping it out there would have kept the per-lane difference the tail removes. Branch operations run the config guard in a branch-operation scope (the split-group and second-value checks; config typed outside the config graph is refused when the data is written).
  • where graph names. A query names no ledger, so "config" and "txn-meta" in a query's where pass through as written, and its fromNamed names stay aliases (even under @base). The ["graph", …] array form is unchanged (JSON-LD where: a compact IRI in ["graph", …] is not expanded, so the pattern silently matches nothing #1993; see Follow-ups).
  • where "default" in an update. When a top-level graph or from gives the where another default graph, a where node's "@graph": "default" reads the ledger's default graph as GRAPH <urn:default>, the name every update reads the default graph by (fix(query): 4xx for unknown graphs and commit refs; urn:default names the default graph #2005), and a fromNamed that restricts the named graphs does not hide it: the update adds urn:default to them for that read (an explicit urn:default there is read only when listed, as on main). A fromNamed alias spelled urn:default that names another graph is refused, wherever it is listed: the name means the default graph before any alias, so such an alias could never be read (on main it takes the name over, and a ["graph", "urn:default", …] reads the alias's graph). Listing urn:default itself (fix(query): 4xx for unknown graphs and commit refs; urn:default names the default graph #2005's form; a listed graph's implicit alias is its own name) is not such an alias. A graph an earlier version registered as urn:default does not take the name from the default graph. Otherwise the where's own default graph is the ledger's, and nothing is added.
  • Subqueries. A ["query", …] subquery takes the naming environment of the where it sits in, in a query too. A query's subquery now sees the query's fromNamed aliases, and its "@graph": "default" inside an enclosing graph pattern is refused, as elsewhere in that where (it used to be accepted there). A subquery's own fromNamed aliases, if any, only add names the parent lacks.
  • dataset_ref.rs is the graph-name half of fix: typed dataset and graph references: one grammar, one resolver #2007's module, item for item (GraphIri(Arc<str>), RefError, GraphSel and its keyword table, with fix: typed dataset and graph references: one grammar, one resolver #2007's text), so landing fix: typed dataset and graph references: one grammar, one resolver #2007 replaces it rather than migrating it.
  • Body options. opts.uniqueProperties is read from the body along with opts.shapes, with the precedence validationMode already has (a programmatic setting wins). A build without SHACL support refuses inline shapes rather than dropping them.
  • Enable recipe. The single-recipe form (OPTIONAL … BIND(COALESCE(?c0, <iri>) AS ?c)) fails on a ledger without config: an IRI produced by BIND cannot be an INSERT template subject. The docs give two recipes, attach to the existing config subject (INSERT … WHERE) and INSERT DATA for a ledger without config; both are tested. The BIND limitation predates this PR (see Follow-ups).
  • vocabulary.md txn-meta example reads with "from": "mydb:main#txn-meta": a node-level @graph in where is now a GRAPH pattern, and resolving a ledger alias as a graph name inside where is separate work.
  • The annotation-scope mutation. The annotation walker no longer carries the update graph key's scope (siblings take the parser's root scope, and one emitter anchors them), so the mutation as written no longer applies. Its remaining counterpart, the top-level routing graph key not being read as a selector, had no test; one was added and the mutation fails it.
  • The no-config mutation. As specified (no config → validate every graph), it cannot fail the explicit-only matrix, whose rows all carry a config; a second mutation (no SHACL group → validate) covers them.
  • Config guard, single-valued set. The eight group pointers (with f:queryDefaults) and every setting the reader takes one value of (listed in config_guard.rs). Multi-valued, not checked: f:policyClass, f:reasoningModes, f:constraintsSource, f:graphOverrides, f:allowedIdentities, f:property, f:ontologyImportMap. Several f:GraphConfig overrides for one graph are reported by the duplicate-graph-override diagnostic rather than refused.
  • Config read failure. Tested through a #[cfg(test)] fault hook (unit tests in tx.rs): config reads fail only on storage errors, which an integration test cannot raise without failing the write for other reasons. The fault hook sits in the config reader itself, so each caller's mapping is under test: config_read_failure_is_config_unavailable_on_every_lane covers a query, the transaction policy context, an insert, a multi-operation update, a fast-forward merge and a push. Each gets ConfigUnavailable, which is HTTP 500 with err:storage/ReadFailure. It is not a 503, because this server reserves 503 for retryable capacity pressure. The consensus flattening carries it as SubmissionError::ConfigUnavailable. The server's mapping of both forms and of the flattened one is tested in a_config_read_failure_is_a_storage_read_failure and in the exhaustive SubmissionError table. AJ chose 500 over 503.
  • Override refusal message. inline SHACL shapes (opts.shapes) refused by this ledger's override control (f:overrideControl f:OverrideNone) for the default graph (or for graph <iri>): no backticks, and the default graph named in words.
  • Shapes-present diagnostic uses the existing type-instance lookup for typed shapes (ledger info only; shape counts are small), then a one-flake probe per SHACL target predicate for untyped ones. A config that sets f:shaclEnabled false anywhere, ledger-wide or for a graph, is a posture rather than a gap, so it is not reported, as a config that enables SHACL anywhere was already not.
  • Graph-sync target label. A malformed sync target is still refused as sync target: … (an existing test pins the label). A graph insert or sync whose payload addresses another graph is refused as payload must not address a graph other than the target (it said "named graphs", even for "default").
  • fluree info prints configDiagnostics for local and tracked ledgers: a local ledger's from config_resolver::diagnose on the loaded ledger (not the full ledger info, whose statistics it does not need; loading is still needed, since the config graph can be in novelty), a tracked ledger's from the server's ledger info. MCP's ledger-info markdown lists them too.
  • Branch-operation refusals. A config refusal on a merge, rebase or revert is carried as its own rejection kind and returned as ApiError::BranchConflict (409), not inferred from a message; a transaction's config refusal stays Parse error, which solo's router keys on.
  • A branch's own system graphs. A merge from a branch the target was created from (directly or not) may carry that branch's config graph writes: they were already an ordinary graph on the target at the fork. Ancestry is read from the nameservice's source_branch chain, only when such a write is staged.
  • The target's own system graphs. A merge may not change the target's config or txn-meta graph at all, assertions or retractions, with what another branch wrote there. The rule goes by the target's graph ids (what the staged flakes route to on the target), not by the name the branch used. Rebase and revert stage the branch's own history and are not covered by it; identifying system graphs by role rather than by IRI would cover them, and is a next question.
  • The recommended config subject. The guard takes urn:fluree:<ledger>:config:ledger (the docs' convention) as the config subject by its IRI, before the config graph holds it; any other subject becomes a config node only when the config graph links or types it. The check is in the authoring scope only, like the typed-outside check, so a branch carrying such data from an earlier release is not refused for it.
  • The expander. To keep annotation placement within budget, the JSON-LD expander changed (commit 33): its position is a borrowed chain whose depth bounds recursion exactly as the old path's length did, collected only for a nested-array error, and the active context is borrowed unless a local or type-scoped context changes it.
  • Plain Turtle inline options. The plain Turtle lane passes the caller's inline uniqueness constraints, inline shapes, requested validation mode and policy context to the check tail, through the same ShaclRequest the TriG fallback builds; it passed none of them, so a plain Turtle document ignored programmatic options that TriG applied.
  • Push graph routing. The binary store is no longer consulted for graph ids; the registry's provisional_ids over every pushed commit's graph_delta gives the ids apply_pushed_commits_to_state assigns.
  • The load-time warning is decided in memory (index stats classes for the shape types and properties for the SHACL target predicates, and POST and PSOT ranges over novelty for the default graph); an index without class or property stats counts as holding no shapes. The config it reads primes the handle's config cache, so the first request does not read it again.
  • GraphQL non-null. Whether sh:minCount makes a field non-null is read from the view's resolved config for its graph (or the ledger's own config when the view was built without it), with no requested mode or identity: the schema describes what the ledger enforces, not what one request asks for. The derived-schema cache keys on the view's graph for that reason.
  • Rebase stops. A rebase stopped by a SHACL violation or a config refusal names the commit it was replaying; a uniqueness violation is returned as its typed error, which has no free-text field for it (Config integrity and SHACL posture: where users see it, the GraphQL schema, and checks that read config two ways #2077).
  • Annotation placement. check_reifiers_match_edges is removed rather than adapted to RDF 1.2 triple terms: rdf:reifies links, triple terms as values, faster annotation queries #2017: placement by construction (@included) makes it unnecessary, and a unit test over ten shapes (both reverse spellings) pins the invariant.

Merging into next

#2017 is merged into next, so the main → next merge after this lands carries a port. git merge-tree against next at 9160de277 reported 11 hunks, all of this PR's (main itself merges into next cleanly): transact/lib.rs (1, the union of the use lists), jsonld.rs (3: the use list, the parse dispatch, the graph insert, which keeps this PR's parse_rooted), edge_annotations.rs (3), expand.rs (1), cli/commands/info.rs (2; keep both next's link-reindex warning and this PR's config diagnostics) and query/parse/mod.rs (1, item 8). The items:

  1. expand.rs: next's triple-term @id arm moves into this PR's expand_node_entries, with this PR's borrowed path.
  2. The parse dispatch: refuse_annotations_under_a_variable_graph reads the f:reifiesSubject slots, so it runs before next's fold_slots_into_links; in the other order it passes vacuously.
  3. The selector-form delete's WHERE needs both next's @reifies form and this PR's node-level @graph. This PR's side alone binds nothing on next (a silent no-op); next's alone drops the selector graph-name resolution.
  4. A triple term under a reverse property is refused in both walkers. Merged as is, the delete walker reaches unreachable!("refused above") (a transactor panic), and the insert walker writes a minted blank node in place of the triple term.
  5. @reifies array blocks 2..n go into the reifier node's @included, as this PR places annotation bundles; next's sibling list is gone here, and bringing it back misplaces them under graph sugar, an update's graph or a named-graph object.
  6. next's annotation-map anchor and its stage.rs strip are dropped (this PR places and emits the graph by construction); re-check item 2 after.
  7. annotation_bundles_land_beside_their_edges and two anchor unit tests are rewritten for next's links; the three annotation integration tests (the edge-graph delete, the selector's graph name, the reverse property) hold as they are.
  8. The CONSTRUCT template filters (the explicit template and the "construct": true shorthand, one hunk each): next keeps TripleTermValue patterns in a template, and this PR's construct_template_pattern keeps graph blocks; the resolved function keeps all three (and recurses into graph blocks for the triple-term values too), and both sites call it.

Items 4 and 5 merge without a conflict marker, so the resolved tree needs tests for them: a triple term under a reverse property, inserted and deleted, and @reifies arrays under graph sugar, an update's graph and a named-graph object (drafted, to run on the resolved tree). grp_triple_terms, it_edge_annotations and this PR's scope suites are the check at the merge. The list was built by git merge-tree and by reading the hunks and call sites, not by a build. Items 1–7 were read at the previous head against next at 0163f7840; at 9160de277 the same files conflicted, plus item 8's. Since then next has merged #1980 (ledger lifecycle; next at 379af5ed0), and git merge-tree of this head against it reports conflicts in 14 files: the six above, plus fluree-db-api/src/tx.rs, fluree-db-consensus (caching.rs, lib.rs, local.rs), fluree-db-server/src/routes/transact.rs and three test files (it_config_graph.rs, it_graphql_shapes.rs, it_policy_cross_ledger.rs). Those come from #1980 meeting this PR's transaction, consensus-error and config changes, and are not analyzed here; #2094 tracks the port.

@aaj3f

aaj3f commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Tests by area, and the mutation table

New and changed tests, by area (all in the owning groups): JSON-LD scope, named graphs, blank issuer, keywords and the refusal-prefix table (fluree-db-transact unit tests); graph scope end to end with SPARQL/TriG twins (it_jsonld_graph_scope); txn-meta (it_txn_meta), including the single-object envelope; annotations, including a variable graph and the ledger's own address or urn:default as the update's graph; node-level @graph in where and its name resolution, including "default" under a graph key, under from and under a restricting fromNamed (with the SPARQL spelling), urn:default read as a node's @graph, in ["graph", …] and as a VALUES-bound graph, the urn:default alias refusal (wherever the alias is listed, and not for a listed urn:default), a graph named @default beside such a where, and a graph an earlier version registered as urn:default; a query's "default" and urn:default on a ledger with a union default graph, beside an update's "default", and the union switch refused on a JSON-LD and a SPARQL update (it_union_default_graph); a subquery naming graphs as its where does; three fromNamed name collisions refused and one graph listed twice accepted; from lists holding the default graph, and the by-name urn:default entry kept out of GRAPH ?g; a top-level values IRI in a new namespace through STR and as a graph name, a top-level values IRI in a registered namespace matching and retracting the stored fact, and a bound IRI in a new namespace stored canonically as a subject and as an object, through a JSON-LD where, a SPARQL WHERE and JSON-LD top-level values (it_transact_update); bulk import refusal; config reader and diagnostics, the query group, the LedgerConfig tripwire and the config-vocabulary test included; the explicit-only matrix over SPARQL-written configs; the Turtle, merge and push lanes without config; fast-forward validation, governed and ungoverned, and uniqueness on branch operations and Turtle insert (it_branch_shacl, it_config_graph); inline shapes and unique properties, from options and from the body (it_shapes_inline, the CLI, and over HTTP in override_control_identity); the dropped-model brick and fail-closed pairs (it_shapes_cross_ledger, it_constraints_cross_ledger, it_policy_cross_ledger, it_config_graph); the injected config-read failure (tx::tests); the config guard on JSON-LD, SPARQL, TriG, Turtle and a branch, across transactions, and on merges, fast-forwards and reverts (refusals and accepted cases, with previews), and on the query group; the config-field vocabulary; the documented recipes and examples. bplatz's review changes add: multi-operation requests governed by the state before them and guarded on their result (SPARQL, Cypher, and Transaction, with a nested multi-operation update or Cypher statement); inline uniqueness refused in policy scope (embedded, body, HTTP); the reader-recorded config vocabulary and the parsed repair recipe; the root subject's fields outside the config graph (JSON-LD, SPARQL, both orders); every stranded config node diagnosed; annotation bundles beside their edges over ten shapes, the edge-graph delete, selector graph names and reverse properties; a merge carrying another branch's config refused (FF, general, preview; parent→child accepted); branch-operation refusals as 409; tracked fluree info against a real server; push validating each named graph on an unindexed copy, graph-id separation, and an unregistered graph refused; the load-time warning (its own test binary); and overflow values read from the default graph under a named base graph. The latest changes add: a transaction governed by its begin state for shapes and uniqueness constraints too; inline uniqueness refused in policy scope through the SPARQL pragma and a Transaction, and applied to plain Turtle; the recommended config subject's fields outside the config graph refused on a ledger with no config, diagnosed when written before, and repaired by the second docs recipe run verbatim; a write and a transaction staged before a uniqueness declaration checked against it; a merge changing the target's config or txn-meta graph refused (fast-forward and general, into a governed and an ungoverned target); plain data a branch wrote to its own config graph refused on a fast-forward into an ungoverned target; the expander's exact depth bound and nested-array path; and a chain of scoped contexts whose term IRIs use an outer prefix, in both expansion modes. The review of the previous push adds: a CONSTRUCT template node's own @graph against ["graph", …] and SPARQL's GRAPH block; the "construct": true shorthand with a graph-scoped where node; an untyped shape in the diagnostics and the load warning (in novelty and indexed), and SHACL switched off not reported; a plain Turtle insert with inline shapes and a requested mode (reject and warn, plain and TriG); GraphQL fields nullable without SHACL, and a view of a graph that enables it derived after a view of one that doesn't; a rebase stopped by a config refusal naming its commit, and one stopped by uniqueness; MCP's diagnostics section; a typed reverse term's annotation; the repair recipe on an anonymous split group; every kind of Transaction operation under a deny-all policy; and the fast-forward test's dev write in warn mode, so it holds whether or not dev reads the config (#2069).

Every fix was checked by reverting only the fix (a mutation), watching its tests fail, and restoring:

Mutation Killed by
classifier: bare graph always a graph key; top-level selector treated as envelope graph_term_in_context_is_a_property_not_a_graph_key, top_level_graph_selector_is_a_node_not_an_envelope
nested nodes scoped to the default graph 6 scope unit tests; 7 of 8 end-to-end scope tests
resolver drops the Var arm; "default" keyword dropped variable_graph_in_update_templates_is_var; the default-keyword test
named-graph content treated as a selector 6 named-graph tests
txn-meta keyed on contains_key("@graph") 2 unit + 2 integration
a top-level single node object in @graph read as a selector the doc_shape table, single_bare_reference_envelope_keeps_its_txn_meta, single_bare_reference_envelope_is_refused_like_its_array_form
issuer reset per item; no collision re-parse; annotation-label firewall off issuer unit + integration; user_b_label_never_collides…; firewall test
keyword refusal off; term @reverse forward; @nest off; adapter @reverse off keyword tests (4 mutations)
emit anchor off unit + it_jsonld_graph_scope + 27 existing annotation tests
walker ignores ["graph", g, …] items (re-run with bundles in @included) annotation_bundles_land_beside_their_edges, annotation_bundle_lands_in_and_names_the_edge_graph, annotations_under_a_variable_graph_are_refused (unit); annotations_follow_their_edge_into_the_graph
walker skips an invalid @graph value without scanning it for annotation keywords an_annotation_key_on_a_wrapper_is_still_refused (existing)
(redefined, see Deviations) the root routing graph key read as a selector root_graph_routing_key_does_not_scope_annotations
annotations under a variable graph not refused annotations_under_a_variable_graph_are_refused
an annotated edge under the ledger's own address keeps its reifier anchor annotation_under_a_default_graph_name_lands_in_the_default_graph (re-run after the rename)
the same, under urn:default only (the address still drops it) annotation_under_a_default_graph_name_lands_in_the_default_graph, its urn:default case (the stray anchor fails the transaction with GraphMismatch)
where node-level @graph off node-level where integration test
where name taken as written (no expansion); "default" inside a graph pattern accepted (2 sites); a query's fromNamed aliases not passed node_level_graph_in_where_resolves_names_like_templates (unit and end to end); node_level_graph_in_query_keeps_from_named_aliases
import refusal off import integration test
reader takes an empty group; diagnostics off (2) reader and diagnostics tests
diagnostics: per-graph enablement ignored; duplicate overrides not reported diagnostics_count_shacl_enabled_for_one_graph; diagnostics_report_duplicate_graph_overrides
no config → validate every graph (the heuristic) shapes-only, Turtle, merge and push lanes
no SHACL group → validate explicit-only matrix
remove the enable config from the test helpers (after commit 13) about 70 shacl_tests, 6 it_branch_shacl, GraphQL and Turtle tests; before commit 13 the same tests still pass through the heuristic, so the migration changed no behavior
inline shapes gated on f:shaclEnabled; override gate off; refusal downgraded to a warning; inline pass off; inline pass compiles stored shapes the corresponding it_shapes_inline rows
inline shapes' policy-scope check off (staging; server); config check reads only the ledger-wide policy group embedded policy-scope tests; the HTTP test; inline_shapes_are_refused_under_policy_defaults_for_any_graph
the body's opts.shapes / opts.uniqueProperties not read body_constraints_are_read_from_opts; 3 it_shapes_inline body tests; insert_honors_inline_shapes_in_the_body (CLI)
a build without SHACL support drops inline shapes instead of refusing them inline_shapes_are_refused_without_shacl_support
resolve before the participation check dropped-model tests (SHACL off; schema) and the fail-closed pair
system graphs participate (SHACL; SHACL and uniqueness) config-repair rows (shapes, local graph, constraints)
cross-ledger policy failure treated as no policy the policy counter-test
read error treated as no config config_read_failure_refuses_a_governed_write
schema source resolved before the no-policy shortcut the dropped-schema test
the check tail skips uniqueness 12 tests, including unique_enforced_on_turtle_insert, merge_enforces_uniqueness_like_a_transaction and the uniqueness fast-forward test
a branch-operation preview reports only SHACL rejections merge_enforces_uniqueness_like_a_transaction
fast-forward not validated; no target governed; every target governed; preview skips a fast-forward 3, 4, 1 (fast_forward_into_an_ungoverned_target_is_not_validated) and 3 of the fast-forward tests
the split-group, typed-outside and second-value checks off the split, typed-outside (incl. branch) and second-value tests
the split-group check across transactions: fields for an already-linked group; an edge to fields elsewhere; fields the transaction retracts still counted config_guard_refuses_a_group_split_across_transactions (each)
branch operations skip the guard; a guard refusal counted as a failure to check (previews error instead of reporting); the refusal does not point at the repair config_guard_refuses_a_merge_that_splits_a_group, config_guard_refuses_a_fast_forward_carrying_a_split_group, config_guard_refuses_a_revert_that_undoes_a_repair (each)
a fast-forward into an ungoverned target is never checked config_guard_refuses_a_fast_forward_carrying_a_split_group
every f: field outside the config graph counted as a split group config_guard_merges_a_branch_while_the_target_changes_its_config
every f: predicate outside the system graphs counted as a config field (an annotation write reads the config) config_fields_are_the_reader_vocabulary; the write's outcome is the same either way, so only the vocabulary test can tell
the where's "default" under another default graph not read by name (as GRAPH <urn:default>) update_where_reads_default_by_name, node_level_graph_in_where_resolves_names_like_templates (unit); where_default_is_the_ledger_default_graph, where_default_leaves_a_graph_named_at_default_alone, where_default_reads_the_default_graph_under_from_named, where_default_is_not_a_graph_registered_as_urn_default
the where's read of "default" by name not recorded update_where_reads_default_by_name (unit); where_default_reads_the_default_graph_under_from_named
a restricting fromNamed hides urn:default from that read where_default_reads_the_default_graph_under_from_named
a fromNamed alias spelled urn:default for another graph accepted where_reads_urn_default_as_the_default_graph (each listing order)
the alias rule applied to a listed urn:default (its implicit alias) where_reads_urn_default_as_the_default_graph and where_default_reads_the_default_graph_under_from_named (their listed cases), and #2005's update_positions_read_urn_default_as_the_default_graph
only the last fromNamed entry carrying the alias checked (what the deduplicated alias map shows) where_reads_urn_default_as_the_default_graph (the alias listed before urn:default)
urn:default resolved after the graph registry, not before it where_default_is_not_a_graph_registered_as_urn_default
a query's "default" read by name, as GRAPH <urn:default> (the union no longer widens it) jsonld_where_default_reads_the_union_and_urn_default_the_default_graph
f:queryDefaults missing from the guard's config edges config_guard_checks_the_query_defaults_group; the LedgerConfig tripwire every_setting_group_is_guarded_and_diagnosed
f:unionDefaultGraph missing from the single-valued set (and so from the config fields) config_guard_checks_the_query_defaults_group; config_fields_are_the_reader_vocabulary
the second-value check alone skips f:unionDefaultGraph config_guard_checks_the_query_defaults_group (the second value commits)
f:queryDefaults missing from the diagnostics' group pointers diagnostics_report_an_empty_query_group; the tripwire
an empty query group read as present diagnostics_report_an_empty_query_group
a ["query", …] subquery parsed without the where's naming environment; without its enclosing-graph depth; its by-name read not reported to the where subquery_names_graphs_as_its_where_does (each)
a fromNamed name for two graphs accepted; an entry's IRI not counted as one of its names; a listed urn:default refused as an alias from_named_names_must_each_mean_one_graph (the first two); #2005's update_positions_read_urn_default_as_the_default_graph, where_default_reads_the_default_graph_under_from_named, where_reads_urn_default_as_the_default_graph (the third)
"default" dropped from a from list; .all → .any in where_reads_ledger_default (RM3); the by-name urn:default entry made enumerable (RM6) from_lists_and_by_name_reads (each)
f:defaultLanguage deleted from the single-valued list (RM7) every_config_vocabulary_term_is_a_field_or_listed
opts.unionDefaultGraph ignored on a JSON-LD update updates_refuse_the_union_switch
an assertion stores a fallback-encoded Sid as bound where_bound_iri_in_a_new_namespace_is_stored_canonically
a top-level values IRI with a code the ledger lacks enters the where as encoded; every top-level values IRI converted, registered ones too top_level_values_iri_reads_as_written_through_str, top_level_values_graph_name_reads_its_graph, where_reads_urn_default_as_the_default_graph; top_level_values_iri_in_a_registered_namespace_matches_as_stored
each operation governed by its own predecessor's state; the guard run per operation; no guard on the merged result multi_operation_update_is_governed_by_the_state_before_it and …_for_uniqueness; multi_operation_update_is_guarded_on_its_result (each)
a Transaction operation, or its Cypher statement, staged without the transaction's governance; a nested stager with its own governance; a nested stager guarding its own result transaction_is_governed_by_the_state_it_began_on (the first three); transaction_is_guarded_on_its_result (the last, and the first)
any f: predicate counted as a config setting on a fast-forward fast_forward_into_an_ungoverned_target_checks_only_config_settings
inline uniqueness not refused in policy scope inline_unique_properties_are_refused_in_a_policy_scoped_request
the root config subject not a config node; stranded fields reported for the root only; f:defaultLanguage dropped from the vocabulary; the recipe misses f:queryDefaults; the recipe has no root branch config_guard_refuses_fields_of_the_config_subject_outside_the_config_graph; diagnostics_report_every_stranded_config_node and the recipe test; the_vocabulary_is_what_the_reader_reads; the_repair_recipe_follows_every_config_edge; repair_recipe_restores_a_split_config (each)
annotation bundles placed in the default graph annotation_bundles_land_beside_their_edges
a delete's object @graph wins; a reverse term read forward; the selector WHERE in the array form annotated_edge_delete_addresses_the_edge_graph_not_the_object_graph; annotation_on_a_reverse_property_reifies_the_stored_edge; selector_annotation_delete_resolves_its_graph_name. Expanding the selector's name before the node-level @graph survives as an equivalent mutant under these contexts
another branch's config graph carried by a merge; the ancestry exception off a_merge_carrying_config_written_on_another_branch_is_refused, fast_forward_is_validated_against_the_config_it_merges_into; the parent→child case
a branch-operation config refusal returned as a check failure (400) config_guard_refuses_a_merge_that_splits_a_group, …_fast_forward_carrying_a_split_group, a_merge_carrying_config_written_on_another_branch_is_refused (409 asserts)
tracked fluree info without the diagnostics info_of_a_tracked_ledger_prints_config_diagnostics
push: no IRI for graphs the commits register; new graph ids counted from the config slot; an unregistered graph given an id push_validates_each_named_graph_under_its_own_config; that and pushed_graphs_are_validated_apart_from_the_ledgers_other_graphs; push_refuses_a_graph_nothing_registers (each). With the old routing restored, the first two fail (the second as a false refusal) and the third gets a 500
the policy context's config read reported as a configuration error; the staging read as a Parse error config_read_failure_is_config_unavailable_on_every_lane (both); config_read_failure_refuses_a_governed_write (the second)
the server's status arm or @type arm for a config read failure off; the consensus flattening untyped (an Execution 500); the server's unflattening untyped a_config_read_failure_is_a_storage_read_failure and submission_error_variants_are_exhaustively_classified (the first two); execution_failure_routes_novelty_by_drainability; submission_error_variants_are_exhaustively_classified
the load-time warning off; its stats gate off; its novelty gate off a_ledger_with_unenforced_shapes_warns_once_per_load (each)
a GRAPH scope's exit decodes only when the outer scope spans several graphs insert_where_reading_the_default_graph_under_using_copies_its_overflow_value, jsonld_update_in_a_named_graph_reading_the_default_graph_copies_its_overflow_value
a transaction's shapes and constraints read from each operation's own base (the config still from the begin state) transaction_is_governed_by_the_state_it_began_on (the shape and f:enforceUnique cases)
the SPARQL pragma not lowered into the options; a Transaction operation staged without the request policy inline_unique_properties_are_refused_in_a_policy_scoped_request (the pragma and transaction cases)
plain Turtle drops the inline constraints; plain Turtle drops the policy context inline_unique_property_applies_to_turtle_with_and_without_graph_blocks (each)
the recommended-subject refusal off; its diagnostic off; the second recipe's path missing a config edge config_guard_refuses_the_conventional_config_subject_outside_the_config_graph (the first two); the_repair_recipe_follows_every_config_edge
a uniqueness declaration not a restage barrier a_stage_behind_a_commit_that_declares_a_property_unique_is_checked_against_it, a_transaction_behind_a_commit_that_declares_a_property_unique_is_checked_against_it
a merge may change the target's system graphs; a fast-forward into an ungoverned target skips a txn-meta write a_merge_changing_the_targets_system_graph_from_another_branch_is_refused (each)
the expander path without its reversal; the depth bound off by one; the depth not counted nested_sequence_reports_its_path; nesting_ceiling_is_exact; the deep-nesting tests
a scoped context's compact term IRIs left unresolved (the fix reverted) scoped_context_terms_expand_against_the_active_context
a fast-forward's config-settings test without the other branch's system-graph arm a_merge_carrying_config_written_on_another_branch_is_refused (plain data in a branch's own config graph, into an ungoverned target)
a CONSTRUCT template drops a node-level graph block jsonld_node_level_graph_in_a_template_names_the_output_graph
untyped shapes not looked for by the diagnostic; SHACL switched off reported; the load gate ignores target predicates in the index stats; in novelty diagnostics_report_shapes_without_enforcement (the first two); a_ledger_with_unenforced_shapes_warns_once_per_load (the last three)
a plain Turtle insert drops inline shapes and the requested mode; the inline pass off inline_shapes_apply_to_turtle_with_and_without_graph_blocks, inline_shapes_honor_a_requested_warn_mode (the first); inline_shapes_honor_a_requested_warn_mode (the second, which only checked that the write committed before)
GraphQL treats every ledger as enforcing SHACL; the schema cache ignores the view's graph min_count_makes_a_field_non_null_only_where_shacl_is_enforced, min_count_follows_the_views_graph; min_count_follows_the_views_graph
a rebase's config refusal without the commit it was replaying rebase_refused_by_a_config_check_names_the_commit
MCP's ledger info without the diagnostics section; local fluree info without the diagnostics test_format_data_model_config_diagnostics; info_prints_config_diagnostics
the shorthand keeps only top-level triples (commit 59's fix reverted) jsonld_construct_shorthand_keeps_a_where_nodes_graph
a request's operations each read a model ledger's current head (commit 61's pins removed); a rollback's restage drops the pins a_transaction_reads_a_model_ledger_at_one_head (each)

@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

While working on this branch since the last push, I decided on several revisions, improvements and fold-ins. They're all reflected in the updated description, but here they are in one place, by area, each with its commit and its tests. The branch is also rebased onto main at c5874d586 (#2006, #2008, #2011, #2013, #2018, #2020, #2022–#2025, #2036 and #1451); how each of those meets this PR is in the description's Sequencing section. With #2008's RDF-text lane in, a Turtle upsert in a multi-operation request or a Transaction is staged with the request's governance and policy like the other lanes (b9f04e45d, in the rebase; every_kind_of_operation_is_checked_against_the_policy, e693eed68); #2108 lands the same change on main on its own.

Graph names in an update's where

  • A where node's "@graph": "default" resolves to urn:default, the name every update reads the default graph by, so the sentinel machinery it used before is gone (2d5824671, query/transact: a where's "default" is the ledger's default graph; tests in it_default_graph_name.rs). The same commit renames names_ledger on the rebased base.
  • A ["query", …] subquery inside an update's where inherits the update's naming environment, so "default", "config" and "txn-meta" resolve there as in the outer where (same commit; subquery_names_graphs_as_its_where_does in it_jsonld_graph_scope).
  • The docs now say what "default" means in each place: under a union default graph, "default" in a query is the union and urn:default is graph 0 alone; in an update both are graph 0.

One config vocabulary, and the guard and diagnostics built on it

  • config_vocab.rs is now the one definition of the config vocabulary: the setting groups, their pointers, the fields of each group, the config classes, and is_config_field. The guard, the diagnostics, graphs_with_config_fields, carries_config_settings and the repair recipe all derive from it (efef2af91, api: one config vocabulary for the reader, the guard, diagnostics and the recipe). feat(query): union default graph, per ledger and per query #2004's query group (f:queryDefaults / f:unionDefaultGraph), which arrived with the rebase, is derived rather than listed, so it's in the guard, the diagnostics and the repair recipe like the others (config_guard_checks_the_query_defaults_group).
  • Three tests keep the vocabulary honest from both sides:
    • the_vocabulary_is_what_the_reader_reads records every predicate the reader looks up while it resolves a maximal config, and checks equality with the vocabulary both ways;
    • every_config_vocabulary_term_is_a_field_or_listed checks the other direction against the f: terms;
    • the_repair_recipe_follows_every_config_edge parses the docs recipe and checks that it covers every edge.
  • Fields of the root f:LedgerConfig subject written outside the config graph are refused like a group's fields, in JSON-LD and SPARQL and in either order, and the repair recipe now also starts from the root (same commit; config_guard_refuses_fields_of_the_config_subject_outside_the_config_graph, and repair_recipe_restores_a_split_config, which runs the docs recipe verbatim).
  • On a ledger with no config yet there's no config subject to compare against, so the recommended subject IRI decides: config fields on urn:fluree:<ledger>:config:ledger written outside the config graph are refused, config or not (7e8ed57dc). Ledger info reports such fields already written as stranded-fields, and the docs give a second repair recipe that starts from that subject and types it in the config graph as its fields move. Tests: config_guard_refuses_the_conventional_config_subject_outside_the_config_graph (JSON-LD and SPARQL, plus the diagnostic on data written before the check), and a fifth case in repair_recipe_restores_a_split_config that runs the second recipe verbatim.
  • Diagnostics walk every node reachable from the config root, so a stranded per-graph override, a partly stranded group and an override without a target are each reported, and the reader's two per-resolve warnings are throttled (efef2af91; diagnostics_report_every_stranded_config_node). The repair recipe handles the two-level case, under the same recipe test.
  • The guard reads config only for config writes (b15857a1a, api: the config guard reads config only for config writes, and b9f04e45d). A plain data write reads nothing beyond its own staged flakes. A write of f:policyClass on an identity outside the config graph reads the config graph once, to learn whether that subject is a config node; the config graph is small, and the read is skipped entirely when the ledger has no config data. The guard's doc comment says exactly this.

Multi-operation requests and transactions

  • The config checks run once, on the request's result, so INSERT DATA {…} ; DELETE DATA {…} is judged by what it commits, not by operation order (b9f04e45d, api: governance is read once, from the state before the write; multi_operation_update_is_guarded_on_its_result).
  • Every operation of a multi-operation SPARQL update, every clause of a Cypher write, and every write staged in a Transaction (feat(api): multi-write transactions #2023) resolves SHACL, uniqueness and config from the state before the request, as a single transaction does. That includes a multi-operation update or Cypher statement nested in a transaction, through SequentialStager::stage_with. It holds in both directions: an earlier operation that turns enforcement off, or removes a shape or constraint, does not admit a later one's data, and one that turns enforcement on, or adds a shape or constraint, does not apply it to the same request (a shape and data violating it, written in one multi-operation update, both commit; main refused that).
  • A transaction staged again after another commit landed is staged against the new head, under the governance and policy the ledger has then. A commit that declares a property unique (f:enforceUnique) now forces that restage too, as a commit that changes the shapes, the class hierarchy or a named graph already did, so the write is checked against the declaration rather than re-based over it (fd07fa2de, api: a commit that declares a property unique makes earlier stages stage again; a_transaction_behind_a_commit_that_declares_a_property_unique_is_checked_against_it).
  • Explicit Bolt transactions commit per statement, so each statement is governed by the state before it, as separate transactions are.
  • Tests: multi_operation_update_is_governed_by_the_state_before_it, …_for_uniqueness, transaction_is_governed_by_the_state_it_began_on (config, a shape removed, f:enforceUnique removed and added) and transaction_is_guarded_on_its_result.

Inline uniqueness constraints

  • opts.uniqueProperties follows the same policy-scope rule as inline shapes: it's refused (400) in a policy-scoped request, on the embedded API, the CLI and the server (c602b2700, api/server: inline uniqueness follows the inline-shapes policy rule). Tests: inline_unique_properties_are_refused_in_a_policy_scoped_request (programmatic options, body options, the SPARQL # PRAGMA unique-properties, and a Transaction under request governance) and …_over_http.
  • A plain Turtle insert now honors programmatic inline constraints as TriG does, refused in policy scope the same way (fdaa4fdd5; inline_unique_property_applies_to_turtle_with_and_without_graph_blocks).

Branch operations and the system graphs

  • A merge, rebase or fast-forward that would carry what a branch wrote to its own config or txn-meta graph into another branch is now refused, with a message that says so: those graphs configure the branch that wrote them, and on the target they'd be ordinary graphs. Merging from a branch the target was created from is unaffected, because that branch's system graphs were already ordinary graphs on the target at the fork (cfaef222e, merge/rebase/revert: refuse carrying another branch's config graph; a_merge_carrying_config_written_on_another_branch_is_refused, fast-forward and general, preview, parent→child still accepted).
  • The other direction is refused too: a merge doesn't change the target's own config or txn-meta graph with what a branch wrote there, where that IRI names an ordinary graph (a613847f4; a_merge_changing_the_targets_system_graph_from_another_branch_is_refused, fast-forward and general, a setting replaced or deleted, txn-meta, into a governed and an ungoverned target). A branch's config is changed on that branch.
  • Identifying system graphs by role rather than by IRI would let such writes merge as config, but it changes graph identity, so it's a next question.

Push

  • Push routes every graph through the ledger's graph registry: registered graphs keep their ids, and graphs the pushed commits register get the ids applying them assigns. Each named graph a push writes is validated under its own config, and a pushed flake in a graph that nothing registers is refused (422) before anything is written (770b4840e, push: graph ids come from the ledger's graph registry).
  • Tests: push_validates_each_named_graph_under_its_own_config (7 cases on an unindexed copy), pushed_graphs_are_validated_apart_from_the_ledgers_other_graphs and push_refuses_a_graph_nothing_registers. push_validates_only_where_config_enables_shacl now pushes between two instances of the same ledger.

Status codes

  • A branch operation the config checks refuse is now a 409 (ApiError::BranchConflict), carried as its own rejection kind rather than inferred from a message. Authoring refusals stay 400 Parse error:.
  • A failure to read the config is one typed error with one status on the write lanes, branch operations, push and queries: TransactError::ConfigUnavailable on the staging lanes (push included), and ApiError::ConfigUnavailable for the policy context and a query's config. It's no longer a Parse error, a 422, or Invalid configuration. The status is HTTP 500 with the existing err:storage/ReadFailure code, on every lane: it's a storage failure, and this server reserves 503 for retryable capacity pressure. The consensus path keeps the code with a typed SubmissionError::ConfigUnavailable (879ae9cdd, api/cli: typed config errors and statuses; tracked info prints diagnostics; config_read_failure_is_config_unavailable_on_every_lane, with the fault injected in the config reader itself).

Fast-forward cost

  • carries_config_settings now matches config fields (the shared vocabulary) and the target's config graph, not any f: predicate, so edge annotations no longer count, and the target's config is read once and passed through the checks (b9f04e45d; fast_forward_into_an_ungoverned_target_checks_only_config_settings). ca4c60954 narrows it further: the f: type arm, which no refusal needs, is dropped, and the other-branch arm gets the test it lacked (plain data a branch wrote to its own config graph, fast-forwarded into an ungoverned target).
  • On a fresh process, an unindexed target still has to be loaded to learn that it governs nothing: measured locally at about 167 ms against about 16 ms on main, for 1,000 target commits. Parity needs the config to be readable without a load, which changes a stored format, so it goes to next: Persisted config summary, so a fast-forward doesn't load an unindexed target (next) #2093.

Enabling SHACL, and a warning at load

  • The docs and the compatibility note lead with the INSERT DATA recipe for a ledger that holds shapes and no config (docs in d8a5f9fdb).
  • A ledger in that state logs a warning once when it loads, not per write, and the check reads the ledger only when its index stats or novelty show a shape (bcf3b2496, api: a ledger warns at load when it holds shapes no config enforces; it_unenforced_shapes_warning, its own binary: unindexed and indexed, once per load, and not for an enforced ledger or one without shapes). An untyped shape (a subject with only a SHACL target) counts too, and a config that switches SHACL off is not reported (f25bfb626).

fluree info on a tracked ledger

  • Tracked mode prints the config diagnostics from the server's ledger info, as local mode does (879ae9cdd; info_of_a_tracked_ledger_prints_config_diagnostics, the CLI binary against a real server over HTTP, and info_prints_config_diagnostics, local).

Annotations

  • I kept the annotation work minimal. What remains: the single f:reifiesGraph emitter, the anchor drop for the ledger's own address, and bundles placed with their edge by construction.
  • An annotation bundle now goes in its edge subject's @included, so it lands in the edge's graph by construction, and the run-time cross-check (check_reifiers_match_edges) is gone; nothing is left in this PR for RDF 1.2 triple terms: rdf:reifies links, triple terms as values, faster annotation queries #2017 to conflict with on that point (4c5e4c050, transact: annotation nodes are included in their edge's subject; annotation_bundles_land_beside_their_edges, 10 shapes, both reverse spellings).
  • In the same commit:
    • a delete of an annotated edge uses the subject's scope, as the insert walker does, so an object's @graph no longer decides the edge's graph (annotated_edge_delete_addresses_the_edge_graph_not_the_object_graph, by id and by selector);
    • a selector-form delete's WHERE uses the node-level @graph, which resolves names as templates do (selector_annotation_delete_resolves_its_graph_name: "default", a dataset alias, a compact IRI);
    • one edge-key classifier, shared by the insert and delete walkers, tells a keyword, a reverse term and a forward term apart, so a reverse term annotates the triple as stored, object → subject (annotation_on_a_reverse_property_reifies_the_stored_edge, term and keyword, insert and delete).
  • Nesting the bundle made the JSON-LD expander's per-level path copies show up in the parse of annotated documents. The commit before it (2ee04b883, json-ld: expansion borrows its path and its context) makes descending a level allocation-free and borrows the context. That fix is independent of the placement and speeds up main by itself, and against a main with the same fix, the default-graph annotation parse is +10.3% and an end-to-end insert −2.0% (the description's Performance section has the table). The main → next port for RDF 1.2 triple terms: rdf:reifies links, triple terms as values, faster annotation queries #2017's annotation work is listed in the description too.
  • While in the expander, a scoped context's term IRIs now expand against the active context (57d373ebf, json-ld: a scoped context's term IRIs expand against the active context; scoped_context_terms_expand_against_the_active_context). A property- or type-scoped context was parsed on its own when its term was defined, so a term IRI written as a compact IRI over a prefix only the outer context defines stayed compact. When a scoped context is applied, its terms' @id, @reverse and datatype IRIs over a prefix it doesn't define now expand against the active context before its terms take effect, as JSON-LD 1.1 processes a scoped context where it's applied. This predates the expander change above.

Follow-ups

The follow-ups are filed, grouped under one tracking issue, #2095, and the description's Follow-ups section links each one.

Data written by earlier releases

No stored format changes, and no migration. The compatibility notes add two behaviors on earlier data:

  • a config with two values for one setting-group pointer now reads the non-empty group, and diagnostics flag it as ambiguous-pointer;
  • edges written through a reverse-defined term were stored as if the term were not reversed, so the same document no longer reaches them. The notes include a repair query.

Happy to talk through any of these.

@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

This also fixes #1991, which the description doesn't reference.

Thanks for confirming all three cases. Fixes #1991 is in the description now.

@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Filed #2069: a branch created from a ledger that has no index yet reads none of its source's ledger config.

Thanks for filing it. It's listed with the breaking change in the description (on a branch created from an unindexed ledger, SHACL is off entirely until #2069 lands, where the shapes-exist fallback used to enforce it), and the description has Follow-up: #2069. fast_forward_into_a_shacl_enabled_target_is_validated relied on that branch behavior, as you noted, so its dev write now asks for warn mode: it commits on dev whether or not dev reads the config, and the fast-forward into main is still refused (c681587c5).

@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

On the review's general points:

aaj3f added 2 commits October 13, 2026 14:43
Brings in #2019, #2044, #2048, #2049 and #2052 (main at 58b5323).

#2052 keys cross-ledger governance by the model ledger's head commit and
opens the model at that commit. Two files conflicted:

- tx.rs: this branch moved the stored-shapes pass into its own block
  (`stored_cache`), which #2052 edited in its old place. The block keeps
  this branch's shape, and its compiled-SHACL cache key carries #2052's
  commit id beside the model's t, as main's does. The shapes model opener
  and the cache key type merged without conflict.
- it_policy_cross_ledger.rs: both sides appended tests; both are kept.
A multi-operation request is governed by the state before it: each
operation reads the config, the shapes and the uniqueness constraints
from the ledger as the request found it. A cross-ledger model ledger
was the exception: each operation resolved its own context, so one that
ran after the model committed read the model's newer head. #2052 pins
a model's head (its `t` and commit) within one resolution context; the
request now carries those pins across its operations, so every operation
reads a model ledger at the head the request's first lookup of it found,
and a model that is the data ledger itself from the state before the
request. A `Transaction` staged again from its own base (after a failed
operation, or to roll back to a savepoint) keeps them; staged again over
a newer head, it is governed as that head is. The SPARQL, Rust API and
compatibility pages say so.

Test: a_transaction_reads_a_model_ledger_at_one_head (a shape the model
activates while a transaction is open, before and after a rollback,
and in a transaction begun after it).
@aaj3f

aaj3f commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Merged main in (e6abd54af), which brings #2052. Its head-commit key is kept as is, including in this PR's moved stored-shapes block.

One commit on top, 0454f401e: a request now reads each model ledger at one head across all of its operations, so a later operation in a multi-operation request or a Transaction can't see a newer model head than the first. That keeps this PR's "governance is read once, from the state before the write" holding together with #2052's key. a_transaction_reads_a_model_ledger_at_one_head failed before it. The description and the tests comment are updated to match.

Brings in #2102 (one HTTP status table), #2068, #2056, #2053, #2046,
#2045 and the v4.2.4 release.

- api error.rs: main's exhaustive status_code / error_type tables, with
  ApiError::ConfigUnavailable and TransactError::ConfigUnavailable
  classified as 500 err:storage/ReadFailure in both, and rows for them
  and RequestOverrideRefused in every_variant.
- server error.rs: main's derivation from ApiError; this branch's
  ConfigUnavailable arms move to the API tables.
- consensus: SubmissionError keeps both ConfigUnavailable and
  CommitContention.
- server routes/transact.rs: both submission arms and test rows.
- transaction.rs, tx_builder.rs: main's txn_attempts loops, with this
  branch's request-governance argument on every stage_plan / stage_all
  call.
@aaj3f

aaj3f commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Merged main in again (cf1d8b586), which brings #2102's single status table. This PR's new errors are now classified there rather than in the server: a config read failure (ConfigUnavailable, API and transact) answers 500 err:storage/ReadFailure in both of ApiError's tables, and a refused request override stays 400 through the generic transaction-error arm. Both have rows in the every_variant table, and the consensus SubmissionError keeps ConfigUnavailable next to #2102's CommitContention. The description's base and gates lines are updated.

@aaj3f

aaj3f commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Merged main back in (b739189) to pick up #2108 and the rest of today's landings. The one conflict was the OpPlan::Rdf arm in fluree-db-api/src/tx_builder.rs, and both sides had made the same change there (this branch also passes the request argument it adds), so the resolved file is byte-identical to this branch's version. #2108's test was already here from e693eed, so it lands once, and it still fails with that argument reverted to None. Nothing else overlaps beyond two auto-merged Cargo.toml/lib.rs lines.

…s root

Both push tests copy a storage root that a dropped instance wrote. The
dropped instance can leave its WAL live, and the WAL's flush thread
deletes a segment once the files it covers are flushed, so a copy running
at the same time could miss the segment (copy file: NotFound on
.fluree-wal/00000001.wal). Disconnecting first retires the log before the
copy starts.
@aaj3f

aaj3f commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

One more small one on top (6f25530, tests only): push_validates_each_named_graph_under_its_own_config failed once under load with copy file: NotFound. It copies a storage root that a dropped instance wrote, and that instance's WAL can still be live, so the flush thread retires .fluree-wal/00000001.wal mid-copy. Both push tests now disconnect().await before copying, as it_wal.rs does; it went from 7/80 failures in parallel runs to 80/80 passing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:transact Staging, commit, import/bulk-import, novelty, retraction semantics breaking-change Backwards-incompatible change; drives the Breaking Changes release-notes section bug Something isn't working as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON-LD update: @annotation inside a ["graph", …] template is written to the default graph, and a top-level graph rejects it

2 participants