diff --git a/BENCHMARKING.md b/BENCHMARKING.md index 3a2cf7cc1e..337ed3e572 100644 --- a/BENCHMARKING.md +++ b/BENCHMARKING.md @@ -72,11 +72,12 @@ this table, so it is the one that rots. Add a row when you add a bench file. | Crate | Bench file | Topic | |---|---|---| -| `fluree-db-api` | `insert_formats.rs` | JSON-LD vs Turtle insert throughput, matrix of (format × txn count × nodes/txn) | +| `fluree-db-api` | `insert_formats.rs` | JSON-LD vs Turtle insert throughput, matrix of (format × txn count × nodes/txn); also Turtle upsert (new subjects and over existing ones) and TriG insert with a trailing graph block | | `fluree-db-api` | `vector_query.rs` | End-to-end vector similarity through the query engine, 1K/5K articles, 768-dim | | `fluree-db-api` | `fulltext_query.rs` | Full-text query through novelty + index | | `fluree-db-api` | `import_bulk.rs` | Bulk Turtle import via `fluree.create(id).import(path).execute()`; single- vs default-threaded | | `fluree-db-api` | `transact_commit.rs` | Single-commit latency on a fresh and a populated ledger (`iter_batched` setup) | +| `fluree-db-api` | `transact_upsert_replace.rs` | Upsert that replaces existing values: default/named graph × novelty/indexed, lang + `@list`, 16-predicate subjects, novelty-heavy base, and the new-subject control | | `fluree-db-api` | `query_cold_reload.rs` | File-backed cold reload (load only, and load + first query) | | `fluree-db-api` | `reindex_full.rs` | `Fluree::reindex(...)` end-to-end against a single-txn populated ledger | | `fluree-db-api` | `reindex_incremental.rs` | Orchestrator's incremental path via `Fluree::trigger_index(...)` over delta novelty | diff --git a/Cargo.lock b/Cargo.lock index d018caacdb..338c6093dd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4007,6 +4007,7 @@ dependencies = [ "tokio", "tracing", "ulid", + "xxhash-rust", "zstd", ] diff --git a/docs/cli/server-integration.md b/docs/cli/server-integration.md index 7e1ca96b46..a215c9e6f3 100644 --- a/docs/cli/server-integration.md +++ b/docs/cli/server-integration.md @@ -349,9 +349,14 @@ MATCH (n:Person {id: 7}) RETURN n Makes one named graph's contents exactly the payload, committing only the delta. Data-bearer auth (same bracket as `/insert` / `/upsert`), not admin. -The CLI converts Turtle to JSON-LD client-side and sends it as -`application/json`; TriG, which that conversion cannot carry, is sent as -`application/trig`. A dry run answers with a delta report and +The CLI sends RDF text as written: Turtle as `text/turtle`, TriG as +`application/trig`, JSON-LD as `application/json`. A server from before +`/sync` read RDF bodies refuses a Turtle body, with `415 Unsupported Media +Type` or (v4.2.0 and v4.2.1) a `400` whose message ends `convert Turtle +payloads client-side`; for that server the CLI converts the Turtle to +JSON-LD client-side, sends it again, and prints a warning. Any other `400` +is reported as is, and TriG is never +converted. A dry run answers with a delta report and must commit nothing; a real run answers with the standard transact response. Designed so the CLI's source of desired contents (today RDF text; later R2RML-mapped Iceberg / CSV / spreadsheet data) is invisible to the server — @@ -1518,7 +1523,7 @@ Content-Type: application/json | `default` (query) | No | Bare key: sync the default graph, said explicitly. Passing it with `graph` is a `400`. The CLI omits `graph` for the default graph. | | `dryRun` (query) | No | `true` → stage and report the delta; commit nothing. | | `allowEmpty` (query) | No | `true` → accept an explicitly empty payload (`"@graph": []`), which clears the graph. Without it an empty payload is a `400`. | -| body | Yes | The graph's desired full contents: insert-shaped JSON-LD, or Turtle / N-Triples / TriG by `Content-Type` (see [Payload formats](../transactions/sync.md#payload-formats)). The CLI sends JSON-LD for JSON-LD and Turtle (converted client-side), so those work against servers that predate RDF bodies; it sends TriG as `application/trig`. Policy headers / `opts` injection follow the [Policy Enforcement Contract](#policy-enforcement-contract). | +| body | Yes | The graph's desired full contents: insert-shaped JSON-LD, or Turtle / N-Triples / TriG by `Content-Type` (see [Payload formats](../transactions/sync.md#payload-formats)). The CLI sends Turtle as `text/turtle` and TriG as `application/trig`, converting Turtle to JSON-LD client-side only when a server that predates RDF bodies refuses it (a `415`, or v4.2.0 / v4.2.1's `400`). Policy headers / `opts` injection follow the [Policy Enforcement Contract](#policy-enforcement-contract). | ### Auth diff --git a/docs/cli/sync.md b/docs/cli/sync.md index c8059b1663..72db3b54bc 100644 --- a/docs/cli/sync.md +++ b/docs/cli/sync.md @@ -42,15 +42,21 @@ produce no flakes, and an identical payload produces **no commit**. History is preserved. See [Sync (graph synchronization)](../transactions/sync.md) for the full semantics, safety rails, and blank-node behavior. -Turtle input is converted to JSON-LD client-side before submission, so a -Turtle export works against any server that implements the `/sync` -endpoint, including servers from before it accepted Turtle bodies. A `.nt` -file is read as Turtle, of which N-Triples is a subset. - -TriG is sent as TriG rather than converted. It is recognized by a `.trig` -name or `--format trig`, and also when a body read as Turtle turns out to -hold graph blocks, as when it is piped in. Its blocks must name the -`--graph` target, and cannot sit beside default-graph triples (see +Turtle input is sent as written and parsed where it is staged, the way +every write reads RDF text (see +[How RDF text is read](../transactions/turtle.md#how-rdf-text-is-read)). +A `.nt` file is read as Turtle, of which N-Triples is a subset. With +`--remote`, a server from before `/sync` accepted Turtle bodies refuses +one: v4.2.0 and v4.2.1 answer 400 (`convert Turtle payloads client-side`), +others 415. For that server the CLI converts the Turtle to JSON-LD, sends +it again, and prints a warning. The conversion loses collection order and +refuses some IRI schemes (`tag:`, `kb:`), so upgrading the server is the +better fix. + +TriG is sent as TriG. It is recognized by a `.trig` name or +`--format trig`, and also when a body read as Turtle turns out to hold +graph blocks, as when it is piped in. Its blocks must name the `--graph` +target, and cannot sit beside default-graph triples (see [payload formats](../transactions/sync.md#payload-formats)): ```bash diff --git a/docs/concepts/datatypes.md b/docs/concepts/datatypes.md index 93875b5c8e..55879f023a 100644 --- a/docs/concepts/datatypes.md +++ b/docs/concepts/datatypes.md @@ -406,11 +406,17 @@ Numeric types (long, double, bigint, decimal) are mutually comparable via automa A literal such as `"2024-02-30"^^xsd:date` or `"300"^^xsd:byte` names a built-in datatype but is not one of its values. JSON-LD transactions and -SPARQL UPDATE reject it, naming the datatype in the error. Turtle +SPARQL UPDATE refuse to insert it, naming the datatype in the error. Turtle transactions and bulk import keep it, as RDF allows: the literal is stored with its text and its datatype and reads back exactly as written, but it is not a date or a number, so it never equals one. +A delete names such a literal as it is stored, so `DELETE DATA` or a JSON-LD +`delete` with the same text and datatype removes it, and one that matches +nothing stored commits nothing. A constant in a WHERE clause or a query is +still refused, so `DELETE WHERE` and `SELECT` reach a stored one only through +a variable (`FILTER(STR(?o) = "2024-02-30")`). + ### Type Casting in Queries SPARQL provides functions for type conversion: diff --git a/docs/concepts/edge-annotations.md b/docs/concepts/edge-annotations.md index 1db70fb81d..2b59c7a203 100644 --- a/docs/concepts/edge-annotations.md +++ b/docs/concepts/edge-annotations.md @@ -27,7 +27,7 @@ If a fact is naturally about a *node* (Alice's birthdate, Acme's industry), put |---|---|---| | **JSON-LD insert / upsert / update** | `@annotation` (or alias `@edge`) on a value object | Most ergonomic. Covers literal-valued edges (with explicit `@type` / `@language`), parallel annotations, named reifiers, body cascades, and **named-graph edges** (an annotation on an edge inside a named graph is written into that same graph, keeping the edge's graph identity). `@reifies` is a query-side construct, **not** an insert form (user-authored `@reifies` on a write is rejected). | | **SPARQL 1.2 UPDATE** | `INSERT DATA { :s :p :o {\| ... \|} }`, `~ `, optional `INSERT { } WHERE { }` templates | Use this when integrating with SPARQL pipelines or when porting from RDF 1.2 / SPARQL-star. **Default graph only:** an annotation tail inside an explicit `GRAPH { }` block, or under a `WITH ` template, is rejected — use the JSON-LD surface or TriG-star for named-graph edge annotations. See [SPARQL 1.2 surface](#sparql-12--rdf-12-surface) below for the per-operation rules. | -| **Turtle / N-Triples / TriG / N-Quads ingest** (`insert`, `upsert`, bulk `import`, graph sync (`fluree sync`, `/sync`), memory import; TriG via `insert` / `upsert` / `import` / `fluree sync` / `/sync`) | RDF 1.2 asserting forms: `:s :p :o ~ {\| ... \|}`, `<< :s :p :o ~ :r >>` in subject or object position, and the canonical `:r rdf:reifies <<( :s :p :o )>>` (the only star spelling N-Triples and N-Quads have) — in the default graph and inside TriG `GRAPH { }` blocks alike | Same on-disk bundle as `@annotation`. An annotation inside a `GRAPH { }` block, or on an N-Quads statement with a graph label, is written into that graph with the edge's graph identity, exactly as JSON-LD `@graph` + `@annotation` does. Every form asserts the base edge (Fluree reifies asserted edges, so `<< s p o >>` and `rdf:reifies <<( s p o )>>` are asserting here). Rejected with a specific error: `<<( ... )>>` anywhere other than the object of `rdf:reifies`, nested triple terms, star constructs inside an annotation body, annotations on collections, and annotations in a TriG `<#txn-meta>` block. Paths that convert to JSON-LD first (`upsert`, `graph sync`, memory import) also reject an annotation on an `rdf:type` edge. See [Turtle ingest](../transactions/turtle.md#edge-annotations-rdf-12--turtle-star). | +| **Turtle / N-Triples / TriG / N-Quads ingest** (`insert`, `upsert`, bulk `import`, graph sync (`fluree sync`, `/sync`), memory import; TriG via `insert` / `upsert` / `import` / `fluree sync` / `/sync`) | RDF 1.2 asserting forms: `:s :p :o ~ {\| ... \|}`, `<< :s :p :o ~ :r >>` in subject or object position, and the canonical `:r rdf:reifies <<( :s :p :o )>>` (the only star spelling N-Triples and N-Quads have) — in the default graph and inside TriG `GRAPH { }` blocks alike | Same on-disk bundle as `@annotation`. An annotation inside a `GRAPH { }` block, or on an N-Quads statement with a graph label, is written into that graph with the edge's graph identity, exactly as JSON-LD `@graph` + `@annotation` does. Every form asserts the base edge (Fluree reifies asserted edges, so `<< s p o >>` and `rdf:reifies <<( s p o )>>` are asserting here). Rejected with a specific error: `<<( ... )>>` anywhere other than the object of `rdf:reifies`, nested triple terms, star constructs inside an annotation body, annotations on collections, and annotations in a TriG `<#txn-meta>` block. The memory import converts Turtle to JSON-LD first and also rejects an annotation on an `rdf:type` edge; the transaction endpoints accept one like any other edge. See [Turtle ingest](../transactions/turtle.md#edge-annotations-rdf-12--turtle-star). | Annotations are backed by a set of [reserved system predicates](../reference/vocabulary.md#edge-annotation-predicates-reserved) that application writes can't author by hand — mint annotations through `@annotation` / `@edge` (JSON-LD) or the RDF 1.2 forms (`~`, `{| |}`, `<< >>`, `rdf:reifies <<( )>>`) in SPARQL UPDATE and Turtle. (Bulk import is an administrative bootstrap path that may ingest already-lowered annotation bundles without up-front validation; see the [storage-internals design doc](../design/edge-annotations.md) for how malformed bundles are handled.) diff --git a/docs/ledger-config/README.md b/docs/ledger-config/README.md index cbb45a8a1e..ee9f3d0693 100644 --- a/docs/ledger-config/README.md +++ b/docs/ledger-config/README.md @@ -207,9 +207,8 @@ User queries against the config graph go through **policy enforcement**. If `f:d The config graph is written and queried through normal CLI transaction and query commands: ```bash -# Write config via SPARQL UPDATE. TriG through `fluree insert` works too, but -# not with the anonymous blank nodes (`[ … ]`) config settings are usually -# written as (#1930); see "Writing from the CLI" in writing-config.md. +# Write config via SPARQL UPDATE. TriG through `fluree insert` or +# `fluree upsert` works too; see "Writing from the CLI" in writing-config.md. fluree update --ledger mydb:main --format sparql -e ' PREFIX f: INSERT DATA { diff --git a/docs/ledger-config/writing-config.md b/docs/ledger-config/writing-config.md index 7632f23ffd..f9d86ed668 100644 --- a/docs/ledger-config/writing-config.md +++ b/docs/ledger-config/writing-config.md @@ -33,7 +33,7 @@ GRAPH { ## Writing from the CLI -`fluree insert` and `fluree upsert` read TriG, but the recipe above writes its settings as anonymous blank nodes (`[ … ]`) inside the `GRAPH` block, which those commands do not read yet ([#1930](https://github.com/fluree/db/issues/1930)). Write the config graph with SPARQL UPDATE through `fluree update` instead: +`fluree insert` and `fluree upsert` read the TriG recipe above as written, anonymous blank nodes (`[ … ]`) inside the `GRAPH` block included. The same settings can also be written with SPARQL UPDATE through `fluree update`: ```bash fluree update -l mydb:main --format sparql -e ' diff --git a/docs/security/policy-in-transactions.md b/docs/security/policy-in-transactions.md index 4f537aa2a3..f7fcfd67ee 100644 --- a/docs/security/policy-in-transactions.md +++ b/docs/security/policy-in-transactions.md @@ -81,6 +81,10 @@ Every modification path runs the same `f:modify` policy check on its staged flak Crucially, the policy is checked against the **flakes**, not the operation type. A transaction that retracts a flake the identity can't modify is rejected just like an insert that asserts one. +A write that restates a stored value as it is stages nothing: an upsert of an unchanged value, a DELETE and INSERT of the same value, or an `ADD`, `COPY` or `MOVE` of a fact the destination graph already holds. When the identity cannot view that value, the restatement is still checked against `f:modify`, so it is refused unless the identity may modify the value, whether or not the value is stored. Restating a value the identity can view commits nothing, with no modify check. + +**Modify without view.** Granting `f:modify` on a value without `f:view` does not keep that value confidential from the identity: an identity that may modify a property can tell whether a given value is stored by writing it. Configure such policies with care. + Enforcement is also independent of the **wire format**: the check runs on the staged flakes, so JSON-LD, SPARQL UPDATE, and Turtle / TriG / N-Triples writes are all governed by the same `f:modify` policy. Sending data as Turtle is not a way to bypass write policy. Class targeting on the write side is **exact**: an `f:onClass` modify policy governs *every* flake whose subject is an instance of the class — including properties the class has never carried before. (Read-side class targeting uses the committed class→property index; write-side targeting matches by the subject's classes directly, since a write may introduce class/property combinations no committed data predicts.) diff --git a/docs/transactions/README.md b/docs/transactions/README.md index dc89521fe5..b1f649dead 100644 --- a/docs/transactions/README.md +++ b/docs/transactions/README.md @@ -22,7 +22,7 @@ Adding new data to the database: ### [Upsert](upsert.md) -Idempotent transactions that replace values for supplied predicates: +Transactions that replace values for supplied predicates, idempotent except for values stored with less detail than they were written: - Upsert semantics - Use cases for upsert - Idempotent operations @@ -285,6 +285,7 @@ For idempotent transactions: - Include unique identifiers - Design for retry safety - Use deterministic IRIs +- Write values at the precision they are stored with: `xsd:integer` for integers beyond 64 bits, at most six fractional digits for `dateTime` and `time` (see [Upsert](upsert.md#idempotency)) ## Best Practices diff --git a/docs/transactions/retractions.md b/docs/transactions/retractions.md index 9053a38d56..c7ece6f482 100644 --- a/docs/transactions/retractions.md +++ b/docs/transactions/retractions.md @@ -304,14 +304,35 @@ Remove entities with specific property: ### Idempotent -Retracting a non-existent triple is a no-op: +Retracting a triple that is not stored is a no-op, and a transaction that +retracts nothing else commits nothing: ```text t=1: No triple exists -t=2: DELETE { ex:alice schema:age 30 } - Result: No change (triple didn't exist) +t=2: DELETE DATA { ex:alice schema:age 30 } + Result: nothing is committed; the ledger stays at t=1 ``` +A DELETE names RDF terms exactly. The datatype must match the stored one +(`30` is `xsd:integer`, so it does not retract `"30"^^xsd:int`); a language +tag matches case-insensitively (`"a"@EN` retracts a stored `"a"@en`). A +`DELETE`/`INSERT` of a triple that is not stored therefore inserts it — the +retraction names nothing, so it cannot cancel the insertion (SPARQL 1.1 +Update's `(DS − D) ∪ I`): + +```sparql +DELETE { ex:alice schema:age 30 } INSERT { ex:alice schema:age 30 } WHERE {} +``` + +stores `ex:alice schema:age 30` if it was absent, and commits nothing if it +was present. + +Once a fact is indexed, a DELETE is matched against what the index stores, +which is less than was written for two kinds of value: an integer too large +for 64 bits keeps no XSD subtype, and a `dateTime` or `time` keeps six +fractional digits. A DELETE naming the same big integer under another integer +datatype, or the same instant to the microsecond, retracts such a fact. + ### No Cascading by Default Retracting an entity doesn't automatically retract references to it: diff --git a/docs/transactions/turtle.md b/docs/transactions/turtle.md index 6f1f8e5533..beefac23ed 100644 --- a/docs/transactions/turtle.md +++ b/docs/transactions/turtle.md @@ -33,7 +33,7 @@ Fluree supports Turtle and TriG on different endpoints with different semantics: | `/data` ([Graph Store](../api/graph-store.md) `PUT` / `POST`) | Supported | Supported (one graph per request) | - **Insert** (`/insert`): Pure insert semantics: triples are added and existing values are kept. Turtle is parsed straight to flakes. A TriG body's graph blocks land in their named graphs and a `<#txn-meta>` block becomes commit metadata; a body with graph blocks is read as TriG under either content type. -- **Upsert** (`/upsert`): For each (subject, predicate) pair, existing values are retracted before new values are asserted. Supports TriG with GRAPH blocks for named graph ingestion. +- **Upsert** (`/upsert`): For each (graph, subject, predicate) the document names, the stored values, every language tag and list position included, are retracted and the document's values asserted (see [Upsert](upsert.md#language-tags-lists-and-blank-nodes)). A TriG body's blocks upsert into their graphs. - **Sync** (`/sync?graph=`): The body becomes the named graph's entire contents, committing only the difference; an unchanged body commits nothing. See [Sync](sync.md#payload-formats) for the TriG rules. N-Triples is a subset of Turtle and is accepted wherever Turtle is: as `application/n-triples` over HTTP, and as a `.nt` file or `--format nt` from the CLI. @@ -621,7 +621,47 @@ curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \ From the CLI, `fluree insert -f data.trig` and `fluree upsert -f data.trig` do the same. -**Known limitation ([#1930](https://github.com/fluree/db/issues/1930)):** on `/insert`, `/upsert` and bulk import, a `GRAPH` block's contents are read by a smaller parser that rejects anonymous blank nodes (`[ … ]`) and collections (`( … )`) with an error naming the construct. Triples outside blocks are unaffected. `/sync` and the [Graph Store Protocol](../api/graph-store.md) read block contents with the full Turtle parser, so they accept both; for `/insert` and `/upsert`, use labeled blank nodes (`_:b1`) inside blocks. +A `GRAPH` block's contents are Turtle like the rest of the document: anonymous blank nodes (`[ … ]`), collections (`( … )`) and RDF 1.2 annotations work inside blocks on every transaction endpoint. **Known limitation ([#1930](https://github.com/fluree/db/issues/1930)):** bulk import (`fluree create --from data.trig`) still reads block contents with a smaller parser that rejects `[ … ]` and `( … )` with an error naming the construct; use labeled blank nodes (`_:b1`) inside blocks there. + +### How RDF Text Is Read + +Every transaction endpoint reads Turtle and TriG with the same parser, once, and stores what it reads; nothing is converted to JSON-LD on the way. So the same document stores the same facts through `/insert`, `/upsert`, `/sync` and the Graph Store routes: + +- **Directives apply in document order.** A `@prefix` or `@base` applies to what follows it, in the default graph and in blocks alike; redefining a prefix later does not change the IRIs before it. A block label is resolved under the declarations in force where the block appears. +- **Relative IRIs resolve against the base.** `` resolves against the `@base` in force. With no base, a relative reference in the default graph is a parse error, while one in a `GRAPH` block, as its label or in its contents, is kept exactly as written: ledger configuration names a ledger by its id, and an id such as `org/governance:main` is a relative reference (`f:ledger `). `<#txn-meta>` with no base in force names the commit-metadata block, as always. +- **Literals are stored as written.** A lexical form that does not fit its datatype (`"abc"^^xsd:integer`) is kept, with its declared datatype, on every endpoint, as insert always did. +- **Any IRI scheme works.** `tag:`, `kb:` and other schemes are stored verbatim, however they are spelled. +- **Collections keep their order and repeats**, and `rdf:type` takes a blank node or a literal like any other predicate. +- **Blank-node labels are document-scoped:** `_:b` in the default graph and in two blocks is one node. Each `[ ]` is a new node. +- **Errors point into the document as sent:** a parse error in a block reports its position in the whole document, and a malformed block (nested, unclosed, a directive inside it, a blank-node label) is named as such. + +An upsert scopes a document's blank nodes to its content, so upserting the same document again addresses the same nodes and commits nothing. The scope ignores statement order (anonymous `[ ]` nodes are numbered in document order, so moving statements that contain them relative to each other gives a new scope); any change to the statements, or to how they are written beyond IRI and quote style (`1` vs `"1"^^xsd:integer`, a language tag's case, a repeated statement, a renamed blank-node label), gives a new scope and new nodes, and the previous nodes stay. To find blank-node subjects that nothing references any more: + +```sparql +SELECT DISTINCT ?b WHERE { + ?b ?p ?o . + FILTER(isBlank(?b)) + FILTER NOT EXISTS { ?s ?q ?b } + FILTER NOT EXISTS { GRAPH ?h { ?s2 ?q2 ?b } } +} +``` + +That query lists subjects in the default graph. Blank nodes a TriG upsert wrote inside `GRAPH` blocks live in those graphs; the same check, graph by graph: + +```sparql +SELECT DISTINCT ?g ?b WHERE { + GRAPH ?g { + ?b ?p ?o . + FILTER(isBlank(?b)) + } + FILTER NOT EXISTS { ?s ?q ?b } + FILTER NOT EXISTS { GRAPH ?h { ?s2 ?q2 ?b } } +} +``` + +Both look for references in every graph. A blank-node label names one node across a TriG document, so a node described in one graph can be referenced from another, and a check within its own graph would list it. + +Review the list before deleting: a document can hold unreferenced blank-node subjects on purpose. Deleting an orphaned node leaves the nodes it referenced orphaned in turn, so repeat until the query returns nothing. ### Querying Named Graphs @@ -672,7 +712,7 @@ Fluree assigns internal graph IDs to named graphs: ### TriG with Transaction Metadata -You can combine named graphs with transaction metadata using the special `#txn-meta` graph fragment. This works on both `insert` and `upsert`: +You can combine named graphs with transaction metadata using the special `#txn-meta` graph fragment (with no `@base` in force). This works on both `insert` and `upsert`: ```trig @prefix ex: . diff --git a/docs/transactions/update-where-delete-insert.md b/docs/transactions/update-where-delete-insert.md index f0d2db6318..f037e4e349 100644 --- a/docs/transactions/update-where-delete-insert.md +++ b/docs/transactions/update-where-delete-insert.md @@ -33,6 +33,12 @@ This: 2. Deletes that age value 3. Inserts the new age value +A DELETE retracts only facts that are stored. A template instantiated with a +value the ledger does not hold (a constant that is absent, or a value bound by +a different pattern) retracts nothing, and cannot cancel an INSERT of the same +triple: `DELETE { x } INSERT { x } WHERE {}` inserts `x` when it is absent. +See [Retraction Semantics](retractions.md#retraction-semantics). + ## WHERE clause capabilities The update transaction `where` clause uses the **same pattern grammar as JSON-LD queries**, so you can use rich patterns like OPTIONAL, UNION, FILTER, VALUES, and subqueries. @@ -689,11 +695,11 @@ Updates ALL entities with status="pending". | Feature | WHERE/DELETE/INSERT | Replace Mode | |---------|---------------------|--------------| -| **Granularity** | Property-level | Entity-level | -| **Other properties** | Preserved | Removed | +| **Granularity** | Value-level | Predicate-level (every value, every language) | +| **Other properties** | Preserved | Preserved | | **Conditional** | Yes (with filters) | No | | **Pattern matching** | Yes | No | -| **Idempotent** | Depends on logic | Yes | +| **Idempotent** | Depends on logic | Yes, with one exception (see [Upsert](upsert.md#idempotency)) | | **Use case** | Partial updates | Complete replacement | ## Best Practices diff --git a/docs/transactions/upsert.md b/docs/transactions/upsert.md index 335e741181..8ce8ff4b0d 100644 --- a/docs/transactions/upsert.md +++ b/docs/transactions/upsert.md @@ -1,6 +1,6 @@ # Upsert -Upsert operations provide idempotent transactions by **replacing the values of the predicates you supply** for an entity (matched by `@id`). +Upsert operations **replace the values of the predicates you supply** for an entity (matched by `@id`). Repeating an upsert is idempotent, with one exception for values stored with less detail than they were written (see [Idempotency](#idempotency)). ## What is Upsert? @@ -8,7 +8,7 @@ Upsert operations provide idempotent transactions by **replacing the values of t - If the entity exists: for each predicate present in your payload, retract existing values for that predicate and assert the new value(s) - If the entity doesn’t exist: create it with the supplied triples -This makes upserts safe to retry: sending the same upsert repeatedly produces the same current-state values for those predicates. +This makes upserts safe to retry: sending the same upsert repeatedly produces the same current-state values for those predicates, with one exception for values stored with less detail than they were written (see [Idempotency](#idempotency)). ## HTTP Endpoint @@ -76,8 +76,12 @@ ex:alice schema:email "alice@example.org" ``` **Operations:** -1. Retract ALL existing properties of ex:alice -2. Assert new properties +1. For each predicate in the payload (`rdf:type`, `schema:name`, `schema:email`, + `schema:age`), retract every value ex:alice currently holds for it +2. Assert the payload's values + +A value the payload restates cancels its own retraction, so it does not +appear in the commit. **Flakes:** ``` @@ -86,7 +90,6 @@ ex:alice schema:name "Alice" (retract) ex:alice schema:email "alice@example.org" (retract) # Assertions (t=2) -ex:alice rdf:type schema:Person (assert) ex:alice schema:name "Alice Smith" (assert) ex:alice schema:email "alice.smith@example.org" (assert) ex:alice schema:age 30 (assert) @@ -100,7 +103,43 @@ ex:alice schema:email "alice.smith@example.org" ex:alice schema:age 30 ``` -Note: The `@type` is re-asserted (types are always included in replace). +`rdf:type schema:Person` is unchanged: it is retracted and re-asserted in the +same transaction, and the two cancel. + +### Language Tags, Lists, and Blank Nodes + +The unit of replacement is the whole `(graph, subject, predicate)`: every +current value is retracted exactly as it is stored. + +- **Language tags.** All languages of a predicate are replaced together. An + upsert of `{"schema:name": {"@value": "Alicia", "@language": "es"}}` over + `"Alice"@en` and `"Alice"@fr` leaves only `"Alicia"@es`. To change one + language and keep the others, use + [WHERE/DELETE/INSERT](update-where-delete-insert.md) with a `FILTER` on + `lang(?v)`, or include every language in the payload. +- **Lists.** A `@list` value is replaced as a whole: every stored position is + retracted, including repeated values, and positions the new list restates + cancel. Replacing a list with plain values (or the reverse) works the same + way. +- **Blank nodes.** A blank node in an upsert payload is identified by the + payload itself, so upserting the same document again addresses the same + node, and its values are replaced like any other subject's. An identical + re-upsert of a document with blank nodes commits nothing. A changed + document mints new blank nodes; the old ones stay (see + [Stable blank-node ids](update-where-delete-insert.md#editing-blank-node-structures-stable-_fdb--ids) + for editing a stored blank node in place). For Turtle and TriG the + identity comes from the parsed statements, not their order: IRIs written + in full or as prefixed names, and strings in single or double quotes, + give the same identity. Other differences in how the statements are + written give a new one, and so new blank nodes: a number or boolean + written bare or as a typed literal (`1` vs `"1"^^xsd:integer`), a + language tag's case (`@en-US` vs `@en-us`), a repeated statement, or a + renamed blank-node label (`_:b` vs `_:c`). See + [How RDF text is read](turtle.md#how-rdf-text-is-read) for the details + and a query that lists blank nodes nothing references. +- **Named graphs.** The graph is part of the unit: an upsert into a named + graph replaces that graph's values and leaves the same predicate's values + in other graphs alone. ## Idempotency @@ -112,19 +151,23 @@ Replace mode is idempotent—repeated submissions produce the same result: ``` Result: Entity created. -**Second Submission (t=2):** +**Second Submission:** ```json {"@id": "ex:alice", "schema:name": "Alice", "schema:age": 30} ``` -Result: No actual changes (retracts and re-asserts same values). +Result: nothing is committed. Each value's retraction and re-assertion cancel, +so the transaction stages no flakes and the ledger stays at t=1. This holds +for language-tagged values, lists, and the payload's blank nodes too. -**Third Submission (t=3):** -```json -{"@id": "ex:alice", "schema:name": "Alice", "schema:age": 30} -``` -Result: No actual changes. - -This makes upserts safe to retry. +This makes upserts safe to retry, with one exception. A value stored with +less detail than it was written is not re-upserted as a no-op: an integer too +large for 64 bits written with an XSD subtype such as `xsd:nonNegativeInteger` +(the index keeps no subtype), or a `dateTime` or `time` with digits past the +microsecond (storage keeps six). Re-upserting such a value as written commits +a retraction and an assertion of the same stored fact, and the value can be +lost. Write these values at the precision they are stored with: `xsd:integer` +for big integers, and at most six fractional digits for temporals. Graph sync +behaves the same way. ## Comparison: Insert vs Update vs Upsert @@ -173,7 +216,7 @@ POST /upsert?ledger=mydb:main **Behavior:** - Replaces values **for the predicates you supply** (per subject) - Leaves other predicates unchanged -- Retry-safe/idempotent for the supplied predicates +- Retry-safe/idempotent for the supplied predicates, except for values stored with less detail than they were written (see [Idempotency](#idempotency)) ## Use Cases @@ -338,9 +381,9 @@ All types are replaced together. ## Edge Cases -### Empty Replacement +### Predicates Not in the Payload -Replacing with minimal data removes other properties: +Only the predicates the payload names are replaced: **Before (t=1):** ```json @@ -348,17 +391,15 @@ Replacing with minimal data removes other properties: "@id": "ex:alice", "schema:name": "Alice", "schema:email": "alice@example.org", - "schema:age": 30, - "schema:telephone": "+1-555-0100" + "schema:age": 30 } ``` -**Replace (t=2):** +**Upsert (t=2):** ```json { "@id": "ex:alice", - "@type": "schema:Person", - "schema:name": "Alice" + "schema:name": "Alice Smith" } ``` @@ -366,18 +407,15 @@ Replacing with minimal data removes other properties: ```json { "@id": "ex:alice", - "@type": "schema:Person", - "schema:name": "Alice" + "schema:name": "Alice Smith", + "schema:email": "alice@example.org", + "schema:age": 30 } ``` -Email, age, and telephone are removed. - -### Partial Updates Not Possible - -Replace mode replaces ALL properties—partial updates not supported. - -For partial updates, use [WHERE/DELETE/INSERT](update-where-delete-insert.md). +To remove a predicate's values, retract them with +[WHERE/DELETE/INSERT](update-where-delete-insert.md), or use +[graph sync](sync.md) to make a whole graph equal a document. ## Error Handling @@ -517,12 +555,12 @@ For partial updates, use WHERE/DELETE/INSERT: | Feature | Default Mode | Replace Mode | |---------|--------------|--------------| -| **Behavior** | Additive | Replace all | -| **Existing properties** | Preserved | Removed | -| **Idempotent** | No | Yes | -| **Partial updates** | Yes (with WHERE/DELETE/INSERT) | No | +| **Behavior** | Additive | Replaces each named predicate's values | +| **Predicates not in the payload** | Preserved | Preserved | +| **Idempotent** | No | Yes, with one exception (see [Idempotency](#idempotency)) | +| **Partial updates** | Yes (with WHERE/DELETE/INSERT) | Per predicate | | **Use case** | Adding data | Synchronization | -| **Retry safety** | Requires care | Safe by default | +| **Retry safety** | Requires care | Safe by default, with the same exception | | **Performance** | Fewer operations | More operations | ## Related Documentation diff --git a/docs/troubleshooting/common-errors.md b/docs/troubleshooting/common-errors.md index 71e21b2265..bb61a50e5a 100644 --- a/docs/troubleshooting/common-errors.md +++ b/docs/troubleshooting/common-errors.md @@ -159,6 +159,8 @@ This error fires from the JSON-LD strict compact-IRI guard. A value that *looks* 2. Misspelled or missing prefix in `@context` 3. Intentionally using a bare `prefix:suffix` string as an opaque identifier +Turtle and TriG writes are not read as JSON-LD, so their IRIs, whatever the scheme (`tag:`, `urn:`, `kb:`), never raise this error. + ### Solutions **Add the missing prefix to @context** (most common fix): diff --git a/fluree-db-api/Cargo.toml b/fluree-db-api/Cargo.toml index 4c447e557e..90c5cbd933 100644 --- a/fluree-db-api/Cargo.toml +++ b/fluree-db-api/Cargo.toml @@ -446,6 +446,13 @@ path = "tests/it_minmax_fast_path_fired.rs" name = "it_upsert_skip_fires" path = "tests/it_upsert_skip_fires.rs" +# Standalone (not in grp_transact): asserts on the `delete_witnessed` routing +# stamp via the thread-local span capture, so it must be the only test in its +# process. Same convention as `it_upsert_skip_fires`. +[[test]] +name = "it_delete_witness_routing" +path = "tests/it_delete_witness_routing.rs" + # Standalone (NOT in grp_ledger): sets FLUREE_MAX_GRAPH_SCAN_FLAKES to exercise # the whole-graph-scan cap, then clears it. The limit is read from the process # env per scan and has no programmatic override, so a grouped binary — whose @@ -629,6 +636,10 @@ harness = false name = "transact_filtered_delete" harness = false +[[bench]] +name = "transact_upsert_replace" +harness = false + [[bench]] name = "query_overlay_only_range" harness = false diff --git a/fluree-db-api/benches/insert_formats.rs b/fluree-db-api/benches/insert_formats.rs index 35a1e364a2..b8571aa13d 100644 --- a/fluree-db-api/benches/insert_formats.rs +++ b/fluree-db-api/benches/insert_formats.rs @@ -14,7 +14,14 @@ // // formats: jsonld, turtle, trig (the same Turtle body inside one // `GRAPH { … }` block — exercises the named-graph -// structured path, which Turtle and JSON-LD never touch) +// structured path, which Turtle and JSON-LD never touch), +// upsert_turtle (the Turtle bodies upserted into a fresh +// ledger: new subjects), upsert_turtle_replace (the same +// upserts over a ledger that already holds them: every +// subject's values are read and restated), and trig_mixed +// (TriG insert of each body with its first half in the +// default graph and its second half in a trailing block — +// the documents the streaming Turtle parse stops on) // txn counts: 10, 100 // nodes/txn: 10, 100, 1000 // @@ -82,6 +89,11 @@ struct PregenData { turtle_txns: Vec, /// `turtle_txns` with the body wrapped in a single named-graph block. trig_txns: Vec, + /// `turtle_txns` with the second half of the body in a trailing block. + trig_mixed_txns: Vec, + /// `turtle_txns` with every statement in its own block, under extra + /// prefix declarations. + trig_blocks_txns: Vec, /// Total flakes produced by inserting all transactions (calibrated once). total_flakes: u64, } @@ -99,12 +111,16 @@ fn pregen( let mut jsonld_txns = Vec::with_capacity(txn_count); let mut turtle_txns = Vec::with_capacity(txn_count); let mut trig_txns = Vec::with_capacity(txn_count); + let mut trig_mixed_txns = Vec::with_capacity(txn_count); + let mut trig_blocks_txns = Vec::with_capacity(txn_count); for txn_idx in 0..txn_count { let data = generate_txn_data(txn_idx, nodes_per_txn); jsonld_txns.push(txn_data_to_jsonld(&data)); let turtle = txn_data_to_turtle(&data); trig_txns.push(turtle_to_trig(&turtle)); + trig_mixed_txns.push(turtle_to_trig_mixed(&turtle)); + trig_blocks_txns.push(turtle_to_trig_blocks(&turtle)); turtle_txns.push(turtle); } @@ -135,6 +151,8 @@ fn pregen( jsonld_txns, turtle_txns, trig_txns, + trig_mixed_txns, + trig_blocks_txns, total_flakes, } } @@ -159,6 +177,70 @@ fn turtle_to_trig(turtle: &str) -> String { out } +/// Keep the first half of a Turtle document's statements in the default +/// graph and move the second half into one trailing `GRAPH { … }` block. +/// Statements are the blank-line-separated paragraphs `txn_data_to_turtle` +/// writes, after its `@prefix` lines. +fn turtle_to_trig_mixed(turtle: &str) -> String { + let (prefixes, body): (Vec<&str>, Vec<&str>) = + turtle.lines().partition(|l| l.starts_with("@prefix")); + let body = body.join("\n"); + let statements: Vec<&str> = body + .split("\n\n") + .map(str::trim) + .filter(|s| !s.is_empty()) + .collect(); + let split = statements.len() / 2; + let mut out = String::with_capacity(turtle.len() + 64); + for p in prefixes { + out.push_str(p); + out.push('\n'); + } + for st in &statements[..split] { + out.push_str(st); + out.push_str("\n\n"); + } + out.push_str("GRAPH {\n"); + for st in &statements[split..] { + out.push_str(st); + out.push_str("\n\n"); + } + out.push_str("}\n"); + out +} + +/// Every statement of a Turtle document in its own `GRAPH { … }` block, +/// over ten graphs, with sixteen more prefix declarations than the document +/// uses: the many-small-blocks shape, where per-block parser setup would +/// dominate. +fn turtle_to_trig_blocks(turtle: &str) -> String { + let (prefixes, body): (Vec<&str>, Vec<&str>) = + turtle.lines().partition(|l| l.starts_with("@prefix")); + let body = body.join("\n"); + let mut out = String::with_capacity(turtle.len() * 2); + for p in prefixes { + out.push_str(p); + out.push('\n'); + } + for i in 0..16 { + out.push_str(&format!( + "@prefix x{i}: .\n" + )); + } + for (k, st) in body + .split("\n\n") + .map(str::trim) + .filter(|s| !s.is_empty()) + .enumerate() + { + out.push_str(&format!( + "GRAPH {{\n{st}\n}}\n", + k % 10 + )); + } + out +} + // --------------------------------------------------------------------------- // Insert runners // --------------------------------------------------------------------------- @@ -210,6 +292,29 @@ async fn run_turtle_inserts( ledger } +async fn run_turtle_upserts( + fluree: &BenchFluree, + ledger: BenchLedger, + txns: &[String], + index_config: &IndexConfig, +) -> BenchLedger { + let mut ledger = ledger; + for txn in txns { + let result = fluree + .upsert_turtle_with_opts( + ledger, + txn, + TxnOpts::default(), + CommitOpts::default(), + index_config, + ) + .await + .unwrap(); + ledger = result.ledger; + } + ledger +} + /// TriG goes through the transact builder's `upsert_turtle`: the plain /// `insert_turtle_with_opts` path and the builder's `insert_turtle` fast path /// both skip `GRAPH`-block extraction. The ledger is addressed by alias, so no @@ -419,6 +524,112 @@ fn bench_insert_formats(c: &mut Criterion) { }); }); + // --- Turtle upsert, new subjects --- + group.bench_with_input( + BenchmarkId::new("upsert_turtle", ¶m_label), + &data, + |b, data| { + b.iter(|| { + let alias = next_ledger_alias("ttl-up"); + rt.block_on(async { + let ledger = fluree.create_ledger(&alias).await.unwrap(); + black_box( + run_turtle_upserts( + &fluree, + ledger, + &data.turtle_txns, + &index_config, + ) + .await, + ) + }) + }); + }, + ); + + // --- Turtle upsert over the same data (read + restate) --- + group.bench_with_input( + BenchmarkId::new("upsert_turtle_replace", ¶m_label), + &data, + |b, data| { + b.iter_batched( + || { + rt.block_on(async { + let alias = next_ledger_alias("ttl-up-rep"); + let ledger = fluree.create_ledger(&alias).await.unwrap(); + run_turtle_inserts( + &fluree, + ledger, + &data.turtle_txns, + &index_config, + ) + .await + }) + }, + |ledger| { + rt.block_on(async { + black_box( + run_turtle_upserts( + &fluree, + ledger, + &data.turtle_txns, + &index_config, + ) + .await, + ) + }) + }, + criterion::BatchSize::PerIteration, + ); + }, + ); + + // --- TriG insert: default graph + trailing block --- + group.bench_with_input( + BenchmarkId::new("trig_mixed", ¶m_label), + &data, + |b, data| { + b.iter(|| { + let alias = next_ledger_alias("trig-mixed"); + rt.block_on(async { + let ledger = fluree.create_ledger(&alias).await.unwrap(); + black_box( + run_turtle_inserts( + &fluree, + ledger, + &data.trig_mixed_txns, + &index_config, + ) + .await, + ) + }) + }); + }, + ); + + // --- TriG insert: every statement in its own block --- + group.bench_with_input( + BenchmarkId::new("trig_blocks", ¶m_label), + &data, + |b, data| { + b.iter(|| { + let alias = next_ledger_alias("trig-blocks"); + rt.block_on(async { + let ledger = fluree.create_ledger(&alias).await.unwrap(); + black_box( + run_turtle_inserts( + &fluree, + ledger, + &data.trig_blocks_txns, + &index_config, + ) + .await, + ) + }) + }); + }, + ); + // Collect a single timed run for the summary table. let jsonld_ms = time_jsonld_run(&rt, &fluree, &data.jsonld_txns, &index_config); let turtle_ms = time_turtle_run(&rt, &fluree, &data.turtle_txns, &index_config); diff --git a/fluree-db-api/benches/transact_upsert_replace.rs b/fluree-db-api/benches/transact_upsert_replace.rs new file mode 100644 index 0000000000..e33fe11d1d --- /dev/null +++ b/fluree-db-api/benches/transact_upsert_replace.rs @@ -0,0 +1,384 @@ +//! Upsert staging latency when the payload replaces values the ledger holds. +//! +//! An upsert retracts every current value of each `(graph, subject, +//! predicate)` its payload names, then asserts the payload. That retraction +//! wave runs on every upsert of an existing entity, and nothing else in the +//! suite exercises it: `insert_formats` upserts only new subjects and +//! `transact_commit` measures inserts. +//! +//! ## Scenarios +//! +//! Each scenario upserts one payload into a freshly built ledger. N comes +//! from the scale (below); every subject named is replaced unless stated. +//! +//! 1. `default_novelty` — N subjects × 1 value, default graph, nothing +//! indexed. +//! 2. `default_indexed` — the same, indexed, novelty empty. +//! 3. `named_novelty` — the same in a named graph, nothing indexed. The +//! lane that walked the graph's whole novelty once per slot. +//! 4. `named_indexed` — the named graph, indexed. +//! 5. `lang_list` — each subject holds two language-tagged values and a +//! two-entry `@list`, indexed; the payload replaces both predicates. +//! Language tags and list positions are exactly what the wave must +//! retract as stored. +//! 6. `wide_predicates` — N/4 subjects × 16 predicates, indexed: the +//! per-slot cost shows up here, where one subject costs 16 lookups. +//! 7. `novelty_heavy` — the indexed base of (2) plus 4×N unrelated +//! novelty flakes. A lookup that translates or walks the whole overlay +//! per call (the #1722 floor) turns into a slope here. +//! 8. `new_subjects` — the payload names N subjects the ledger has never +//! seen: the #1549 absence skip, which must stay flat. +//! +//! Every scenario asserts in setup that it reached the lane it names +//! (indexed or not, graph registered, novelty size), and after each +//! measured upsert that the receipt's retract/assert counts are the ones +//! the scenario implies. A fixture that drifts fails instead of reporting +//! a number for the wrong path. +//! +//! ## Matrix +//! +//! inputs: BenchScale → N (Tiny=500, Small=2k, Medium=8k, Large=24k) +//! metric: ns/upsert (criterion default; no Throughput — the guard is +//! on the curve, like `transact_filtered_delete`) +//! +//! ## Running +//! +//! cargo bench -p fluree-db-api --bench transact_upsert_replace +//! cargo bench -p fluree-db-api --bench transact_upsert_replace -- --test +//! FLUREE_BENCH_SCALE=medium cargo bench -p fluree-db-api --bench transact_upsert_replace +//! +//! ## Cargo.toml + budget already wired (see fluree-db-api/Cargo.toml, +//! regression-budget.json). + +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion}; +use fluree_bench_support::{ + bench_runtime, current_profile, current_scale, init_tracing_for_bench, next_ledger_alias, + BenchScale, +}; +use fluree_db_api::{ + CommitOpts, Fluree, FlureeBuilder, IndexConfig, LedgerState, ReindexOptions, TxnOpts, +}; +use serde_json::{json, Value as JsonValue}; + +/// Map BenchScale to N, the number of subjects the upsert replaces. +fn scale_n(scale: BenchScale) -> usize { + match scale { + // Keep tiny tiny so PR-gated runs finish quickly. + BenchScale::Tiny => 500, + BenchScale::Small => 2_000, + BenchScale::Medium => 8_000, + BenchScale::Large => 24_000, + } +} + +const CTX_PREFIX: &str = "http://example.org/ns/"; +const GRAPH: &str = "http://example.org/ns/bench-graph"; +/// Predicates per subject in `wide_predicates`. +const WIDE: usize = 16; +/// Subjects per base-load commit. +const BATCH: usize = 500; + +/// Novelty must survive the fixture: a reindex would fold it into the base. +fn index_config() -> IndexConfig { + IndexConfig { + reindex_min_bytes: 1 << 40, + reindex_max_bytes: 1 << 41, + } +} + +fn ctx() -> JsonValue { + json!({"ex": CTX_PREFIX}) +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Scenario { + DefaultNovelty, + DefaultIndexed, + NamedNovelty, + NamedIndexed, + LangList, + WidePredicates, + NoveltyHeavy, + NewSubjects, +} + +impl Scenario { + const ALL: [Scenario; 8] = [ + Scenario::DefaultNovelty, + Scenario::DefaultIndexed, + Scenario::NamedNovelty, + Scenario::NamedIndexed, + Scenario::LangList, + Scenario::WidePredicates, + Scenario::NoveltyHeavy, + Scenario::NewSubjects, + ]; + + fn name(self) -> &'static str { + match self { + Scenario::DefaultNovelty => "default_novelty", + Scenario::DefaultIndexed => "default_indexed", + Scenario::NamedNovelty => "named_novelty", + Scenario::NamedIndexed => "named_indexed", + Scenario::LangList => "lang_list", + Scenario::WidePredicates => "wide_predicates", + Scenario::NoveltyHeavy => "novelty_heavy", + Scenario::NewSubjects => "new_subjects", + } + } + + fn indexed(self) -> bool { + !matches!(self, Scenario::DefaultNovelty | Scenario::NamedNovelty) + } + + fn graph(self) -> Option<&'static str> { + match self { + Scenario::NamedNovelty | Scenario::NamedIndexed => Some(GRAPH), + _ => None, + } + } + + /// Subjects in the base (and, except for `new_subjects`, in the upsert). + fn subjects(self, n: usize) -> usize { + match self { + Scenario::WidePredicates => (n / 4).max(1), + _ => n, + } + } + + /// Retractions and assertions the measured upsert must commit. + fn expected_counts(self, n: usize) -> (usize, usize) { + let subjects = self.subjects(n); + match self { + // Two tagged values + two list entries out, one of each in. + Scenario::LangList => (4 * subjects, 2 * subjects), + Scenario::WidePredicates => (WIDE * subjects, WIDE * subjects), + Scenario::NewSubjects => (0, subjects), + _ => (subjects, subjects), + } + } +} + +/// One node of the base load or of the measured upsert. +fn node(scenario: Scenario, i: usize, replacement: bool) -> JsonValue { + let id = if scenario == Scenario::NewSubjects && replacement { + format!("ex:new{i}") + } else { + format!("ex:s{i}") + }; + let mut node = match scenario { + Scenario::LangList if replacement => json!({ + "ex:label": {"@value": format!("b{i}"), "@language": "en"}, + "ex:items": {"@list": [format!("z{i}")]}, + }), + Scenario::LangList => json!({ + // Distinct lexical forms per tag, so the count check below + // reads the same whether or not the retractions keep the tag. + "ex:label": [ + {"@value": format!("a{i}"), "@language": "en"}, + {"@value": format!("c{i}"), "@language": "fr"}, + ], + "ex:items": {"@list": [format!("x{i}"), format!("y{i}")]}, + }), + Scenario::WidePredicates => { + let tag = if replacement { "n" } else { "o" }; + let mut obj = serde_json::Map::new(); + for k in 0..WIDE { + obj.insert(format!("ex:p{k}"), json!(format!("{tag}{i}-{k}"))); + } + JsonValue::Object(obj) + } + _ => { + let tag = if replacement { "new" } else { "old" }; + json!({"ex:v": format!("{tag}{i}")}) + } + }; + let obj = node.as_object_mut().expect("object"); + obj.insert("@id".to_string(), json!(id)); + if let Some(g) = scenario.graph() { + obj.insert("@graph".to_string(), json!(g)); + } + node +} + +fn payload(nodes: Vec) -> JsonValue { + json!({"@context": ctx(), "@graph": nodes}) +} + +/// The measured upsert. +fn upsert_payload(scenario: Scenario, n: usize) -> JsonValue { + payload( + (0..scenario.subjects(n)) + .map(|i| node(scenario, i, true)) + .collect(), + ) +} + +/// Everything one measured iteration consumes. The measured routine hands +/// the tempdir and the instance back to criterion, which drops them after +/// the clock stops: deleting the file-backed store costs more than the +/// upsert itself, and its cost follows the filesystem, not the code. +struct Fixture { + db_dir: tempfile::TempDir, + fluree: Fluree, + ledger: LedgerState, +} + +async fn insert_batches( + fluree: &Fluree, + mut ledger: LedgerState, + count: usize, + mut make: impl FnMut(usize) -> JsonValue, +) -> LedgerState { + let mut from = 0usize; + while from < count { + let to = (from + BATCH).min(count); + let r = fluree + .insert_with_opts( + ledger, + &payload((from..to).map(&mut make).collect()), + TxnOpts::default(), + CommitOpts::default(), + &index_config(), + ) + .await + .expect("fixture insert"); + ledger = r.ledger; + from = to; + } + ledger +} + +/// Load the base, index it when the scenario says so, and add the +/// unrelated novelty of `novelty_heavy`. Asserts the lane on the way out. +fn build_fixture(rt: &tokio::runtime::Runtime, scenario: Scenario, n: usize) -> Fixture { + rt.block_on(async { + let db_dir = tempfile::tempdir().expect("db tmpdir"); + let fluree = FlureeBuilder::file(db_dir.path().to_string_lossy().to_string()) + .with_ledger_cache_config(fluree_db_api::LedgerManagerConfig::default()) + .build() + .expect("build file-backed Fluree"); + let alias = next_ledger_alias("tur"); + let ledger = fluree.create_ledger(&alias).await.expect("create_ledger"); + let mut ledger = insert_batches(&fluree, ledger, scenario.subjects(n), |i| { + node(scenario, i, false) + }) + .await; + + if scenario.indexed() { + drop(ledger); + fluree + .reindex(&alias, ReindexOptions::default()) + .await + .expect("reindex"); + ledger = fluree.ledger(&alias).await.expect("reload after reindex"); + assert!( + ledger.snapshot.range_provider.is_some() && ledger.novelty.is_empty(), + "{}: fixture needs an indexed base and empty novelty", + scenario.name() + ); + } else { + assert!( + ledger.snapshot.range_provider.is_none(), + "{}: fixture must stay unindexed", + scenario.name() + ); + } + + if scenario == Scenario::NoveltyHeavy { + ledger = insert_batches( + &fluree, + ledger, + 4 * n, + |i| json!({"@id": format!("ex:u{i}"), "ex:w": format!("w{i}")}), + ) + .await; + assert!( + ledger.novelty.len() >= 4 * n, + "novelty_heavy: the unrelated novelty must survive the fixture" + ); + } + + if let Some(g) = scenario.graph() { + assert!( + ledger.snapshot.graph_registry.graph_id_for_iri(g).is_some(), + "{}: the named graph must be registered before the upsert", + scenario.name() + ); + } + + Fixture { + db_dir, + fluree, + ledger, + } + }) +} + +fn bench_transact_upsert_replace(c: &mut Criterion) { + init_tracing_for_bench(); + + let rt = bench_runtime(); + let scale = current_scale(); + let profile = current_profile(); + let n = scale_n(scale); + + eprintln!(" [transact_upsert_replace] scale={} n={n}", scale.as_str()); + + let mut group = c.benchmark_group("transact_upsert_replace"); + group.sample_size(profile.sample_size()); + // The fixture is rebuilt per iteration because the measured upsert + // changes it; Flat keeps criterion from assuming a cheap, repeatable + // routine. + group.sampling_mode(criterion::SamplingMode::Flat); + + for scenario in Scenario::ALL { + let upsert = upsert_payload(scenario, n); + let (retracts, asserts) = scenario.expected_counts(n); + group.bench_with_input( + BenchmarkId::new(scenario.name(), scale.as_str()), + &n, + |b, &n| { + b.iter_batched( + // Setup: base load (+ reindex, + novelty). NOT measured. + || build_fixture(&rt, scenario, n), + // Measured: one upsert over every subject named. The + // store goes back to criterion to drop untimed. + |fixture| { + let Fixture { + db_dir, + fluree, + ledger, + } = fixture; + let ledger = rt.block_on(async { + let result = fluree + .upsert_with_opts( + ledger, + &upsert, + TxnOpts::default(), + CommitOpts::default(), + &index_config(), + ) + .await + .expect("upsert"); + assert_eq!( + (result.receipt.retract_count, result.receipt.assert_count), + (retracts, asserts), + "{}: unexpected (retracts, asserts)", + scenario.name() + ); + result.ledger + }); + black_box((db_dir, fluree, ledger)) + }, + criterion::BatchSize::PerIteration, + ); + }, + ); + } + + group.finish(); +} + +criterion_group!(benches, bench_transact_upsert_replace); +criterion_main!(benches); diff --git a/fluree-db-api/clippy.toml b/fluree-db-api/clippy.toml new file mode 100644 index 0000000000..e4b14d9c6a --- /dev/null +++ b/fluree-db-api/clippy.toml @@ -0,0 +1,7 @@ +# RDF text is staged from one parse (`fluree_db_transact::parse_rdf_text`). +# Converting it to JSON-LD first loses collection order, refuses IRIs whose +# scheme reads as an undefined prefix (`tag:`, `kb:`), and reads literals +# differently from insert, so no transaction lane may do it quietly. +disallowed-methods = [ + { path = "fluree_graph_turtle::parse_to_json", reason = "lossy JSON-LD round trip: stage RDF text with fluree_db_transact::parse_rdf_text" }, +] diff --git a/fluree-db-api/src/commit_data.rs b/fluree-db-api/src/commit_data.rs index 8a2076902d..f8ccd5b3de 100644 --- a/fluree-db-api/src/commit_data.rs +++ b/fluree-db-api/src/commit_data.rs @@ -126,13 +126,90 @@ where } } Fold::Undo => { + let mut skipped = 0usize; for flakes in &ordered { for flake in flakes { + if is_malformed_retraction(flake) { + skipped += 1; + continue; + } acc.push_newest_first(&flake.invert()); } } + if skipped > 0 { + tracing::warn!( + skipped, + "revert skipped retractions no stored fact could match (an \ + rdf:langString without a language tag, or a tag on another \ + datatype); undoing one would assert an invalid literal" + ); + } } } data.flakes = acc.finish(); data } + +/// A retraction whose shape no stored fact has: an `rdf:langString` +/// without a language tag, or a tag on any other datatype. +/// +/// Upserts written before the retraction resolver rebuilt each retraction +/// from a query binding and dropped the tag, so `"a"@en` was retracted as a +/// tagless `"a"^^rdf:langString`. Such a retraction matched nothing and is +/// inert for reads and indexing, but a revert inverts every flake of the +/// reverted commits, and inverting it asserts a literal that cannot exist. +/// Legacy retractions of list entries without their position are +/// well-formed plain literals, so no shape check can tell them apart. +fn is_malformed_retraction(flake: &Flake) -> bool { + if flake.op { + return false; + } + let lang_string = flake.dt.namespace_code == fluree_vocab::namespaces::RDF + && flake.dt.name.as_ref() == "langString"; + let tagged = flake.m.as_ref().is_some_and(|m| m.lang.is_some()); + lang_string != tagged +} + +#[cfg(test)] +mod tests { + use super::*; + use fluree_db_core::{FlakeMeta, FlakeValue, Sid}; + + fn label(v: &str, dt: &str, lang: Option<&str>, t: i64, op: bool) -> Flake { + Flake::new( + Sid::new(100, "s"), + Sid::new(100, "label"), + FlakeValue::String(v.to_string()), + Sid::new(fluree_vocab::namespaces::RDF, dt), + t, + op, + lang.map(FlakeMeta::with_lang), + ) + } + + /// Reverting an upsert written before the retraction resolver: its + /// retraction of `"a"@en` lost the tag. Inverting that retraction would + /// assert `"a"^^rdf:langString` with no tag, so the undo skips it; the + /// well-formed flakes of the same commit still invert. + #[test] + fn undo_skips_retractions_no_stored_fact_could_match() { + let legacy_upsert = Commit::new( + 2, + vec![ + // The phantom: tagless langString. + label("a", "langString", None, 2, false), + // Well-formed: the upsert's assertion. + label("b", "langString", Some("en"), 2, true), + // A tag on a non-langString datatype is the other malformed + // shape. + label("c", "HTML", Some("en"), 2, false), + ], + ); + let undone = collect_from_commits([legacy_upsert], Fold::Undo).flakes; + assert_eq!(undone.len(), 1, "{undone:?}"); + let f = &undone[0]; + assert!(!f.op, "the assertion of \"b\"@en inverts to its retraction"); + assert_eq!(f.o, FlakeValue::String("b".to_string())); + assert_eq!(f.m.as_ref().and_then(|m| m.lang.as_deref()), Some("en")); + } +} diff --git a/fluree-db-api/src/merge_preview.rs b/fluree-db-api/src/merge_preview.rs index 9bd31dcd30..ab6e91b110 100644 --- a/fluree-db-api/src/merge_preview.rs +++ b/fluree-db-api/src/merge_preview.rs @@ -890,8 +890,8 @@ async fn build_conflict_details( current_asserted_for_key(source_state, &key), current_asserted_for_key(target_state, &key), )?; - let source_values = resolve_flake_list(&source_flakes, source_compactor)?; - let target_values = resolve_flake_list(&target_flakes, target_compactor)?; + let source_values = resolve_fact_list(&source_flakes, source_compactor)?; + let target_values = resolve_fact_list(&target_flakes, target_compactor)?; Ok::<_, ApiError>(ConflictDetail { key, @@ -906,13 +906,13 @@ async fn build_conflict_details( .await } -fn resolve_flake_list( - flakes: &[Flake], +fn resolve_fact_list( + facts: &[fluree_db_transact::StoredFact], compactor: &IriCompactor, ) -> Result> { - flakes + facts .iter() - .map(|flake| resolve_flake(compactor, flake)) + .map(|fact| resolve_flake(compactor, fact.flake())) .collect() } diff --git a/fluree-db-api/src/rebase.rs b/fluree-db-api/src/rebase.rs index 25be4c8962..52c763ff94 100644 --- a/fluree-db-api/src/rebase.rs +++ b/fluree-db-api/src/rebase.rs @@ -8,13 +8,14 @@ use crate::commit_data::{key_of, OwnChanges}; use crate::error::{ApiError, Result}; use fluree_db_core::LedgerId; use fluree_db_core::{ - range_with_overlay, BranchedContentStore, Commit, ConflictKey, ContentId, ContentStore, Flake, - IndexType, RangeMatch, RangeOptions, RangeTest, DEFAULT_GRAPH_ID, + BranchedContentStore, Commit, ConflictKey, ContentId, ContentStore, Flake, DEFAULT_GRAPH_ID, }; use fluree_db_ledger::LedgerState; use fluree_db_nameservice::NsRecordSnapshot; use fluree_db_novelty::{delta_keys_of, FactKey}; -use fluree_db_transact::{CommitOpts, NamespaceRegistry, StagedCommit}; +use fluree_db_transact::{ + CommitOpts, CurrentFacts, NamespaceRegistry, Slot, StagedCommit, StoredFact, +}; use rustc_hash::FxHashSet; use serde::{Deserialize, Serialize}; use std::sync::Arc; @@ -828,12 +829,9 @@ impl crate::Fluree { current_asserted_for_key(source_state, key) .await? .into_iter() - .filter(|flake| !reasserted.contains(&FactKey::of(flake))) - .map(|flake| Flake { - op: false, - t: 0, // restamped by StagedLedger::new - ..flake - }), + .filter(|fact| !reasserted.contains(&FactKey::of(fact.flake()))) + // `t` is restamped by StagedLedger::new. + .map(|fact| fact.retract(0).into_flake()), ); } @@ -931,10 +929,18 @@ fn find_conflicting_keys( .collect() } +/// The facts `state` currently holds for a conflict key's `(subject, +/// predicate, graph)`, graph stamped. +/// +/// Read through [`CurrentFacts`], which stamps the key's graph on index +/// rows. A range read returns an index-resident row with `g: None` whatever +/// graph it lives in, so filtering on `g == key.g` dropped every indexed +/// named-graph value: a take-source merge then left the target's value in +/// place next to the source's. pub(crate) async fn current_asserted_for_key( state: &LedgerState, key: &ConflictKey, -) -> Result> { +) -> Result> { let g_id = match &key.g { None => DEFAULT_GRAPH_ID, Some(g_sid) => match state @@ -946,33 +952,15 @@ pub(crate) async fn current_asserted_for_key( None => return Ok(Vec::new()), }, }; - - let match_val = RangeMatch::subject_predicate(key.s.clone(), key.p.clone()); - let opts = RangeOptions { - to_t: Some(state.t()), - ..Default::default() - }; - - let flakes = range_with_overlay( - &state.snapshot, + let slot = Slot { g_id, - state.novelty.as_ref(), - IndexType::Spot, - RangeTest::Eq, - match_val, - opts, - ) - .await?; - - // The range is scoped to the key's graph, but index rows decode with no - // graph of their own, so each takes the key's rather than being compared - // against it. - Ok(flakes - .into_iter() - .filter(|flake| flake.op) - .map(|flake| Flake { - g: key.g.clone(), - ..flake - }) - .collect()) + g_sid: key.g.clone(), + s: key.s.clone(), + p: key.p.clone(), + }; + Ok(CurrentFacts::new(state) + .of_slots(std::slice::from_ref(&slot)) + .await? + .pop() + .unwrap_or_default()) } diff --git a/fluree-db-api/src/shacl_tests.rs b/fluree-db-api/src/shacl_tests.rs index 1efd5bc8af..764527b3ee 100644 --- a/fluree-db-api/src/shacl_tests.rs +++ b/fluree-db-api/src/shacl_tests.rs @@ -3718,6 +3718,65 @@ async fn shacl_unique_lang() { assert_shacl_violation(err, "more than one value"); } +/// An upsert that replaces every language's label satisfies `sh:uniqueLang`: +/// the replaced values are retracted with their language tags, so the +/// staged state holds one label per language. A second label in a language +/// is still refused. +#[tokio::test] +async fn a_refresh_upsert_satisfies_unique_lang() { + let fluree = FlureeBuilder::memory().build_memory(); + let context = shacl_context(); + let shape_txn = json!({ + "@context": context.clone(), + "@id": "ex:UniqueLabelShape", + "@type": "sh:NodeShape", + "sh:targetClass": {"@id": "ex:Translated"}, + "sh:property": [{ + "@id": "ex:pshape_unique_label", + "sh:path": {"@id": "ex:label"}, + "sh:uniqueLang": true + }] + }); + let ledger = fluree + .create_ledger("shacl/uniq-refresh:main") + .await + .unwrap(); + let ledger = fluree.upsert(ledger, &shape_txn).await.unwrap().ledger; + let doc = |en: &str, fr: &str| { + json!({ + "@context": context.clone(), + "@id": "ex:doc1", + "@type": "ex:Translated", + "ex:label": [ + {"@value": en, "@language": "en"}, + {"@value": fr, "@language": "fr"} + ] + }) + }; + let ledger = fluree + .upsert(ledger, &doc("colour", "couleur")) + .await + .expect("first labels") + .ledger; + let ledger = fluree + .upsert(ledger, &doc("color", "coloris")) + .await + .expect("a refresh replaces each language's label") + .ledger; + let err = fluree + .insert( + ledger, + &json!({ + "@context": context.clone(), + "@id": "ex:doc1", + "ex:label": {"@value": "hue", "@language": "en"} + }), + ) + .await + .unwrap_err(); + assert_shacl_violation(err, "more than one value"); +} + /// `sh:pattern` on IRI values matches the full decoded IRI (SPARQL `STR()`), /// not the SID name fragment — and non-matching IRIs violate. #[tokio::test] diff --git a/fluree-db-api/src/tx.rs b/fluree-db-api/src/tx.rs index 9160ac6a1c..9da3f82859 100644 --- a/fluree-db-api/src/tx.rs +++ b/fluree-db-api/src/tx.rs @@ -11,7 +11,6 @@ use crate::wasm_compat::IndexerHandle; use crate::{ApiError, Result}; use crate::{TrackedErrorResponse, Tracker, TrackingOptions, TrackingTally}; use fluree_db_core::ledger_config::LedgerConfig; -use fluree_db_core::DatatypeConstraint; use fluree_db_core::LedgerId; use fluree_db_core::{ range_with_overlay, FlakeValue, GraphId, IndexType, RangeMatch, RangeOptions, RangeTest, Sid, @@ -25,9 +24,8 @@ use fluree_db_transact::stage_with_graph_delta as stage_txn; #[cfg(feature = "shacl")] use fluree_db_transact::validate_view_with_shacl; use fluree_db_transact::{ - commit as commit_txn, parse_transaction, resolve_trig_meta, CommitOpts, CommitReceipt, - GraphSel, NamedGraphBlock, NamespaceRegistry, RawTrigMeta, StageOptions, TemplateTerm, - TripleTemplate, Txn, TxnOpts, TxnType, + commit as commit_txn, parse_transaction, CommitOpts, CommitReceipt, GraphSel, + NamespaceRegistry, Placement, StageOptions, Txn, TxnOpts, TxnType, }; use fluree_vocab::config_iris; use rustc_hash::{FxHashMap, FxHashSet}; @@ -38,15 +36,14 @@ use serde_json::Value as JsonValue; /// document parsed as Turtle, a fallback when it was TriG. pub const TURTLE_INSERT_SITE: &str = "turtle_insert"; -/// Stable id for an upsert payload, used as its blank-node skolem scope. +/// Stable id for a JSON-LD upsert payload, used as its blank-node skolem +/// scope. (RDF text derives its scope from the parsed statements instead; +/// see [`fluree_db_transact::parse_rdf_text_txn`].) /// /// Streams the JSON through the hasher rather than serializing it to a /// `String` first, so a bulk payload does not pay a second full copy of -/// itself. The TriG blocks fold in their graph IRI, triples and reifier -/// attachments but not their prefix map: prefixes only decide how the triples -/// were expanded, and a `FxHashMap` has no stable iteration order, which -/// would make the scope differ between two runs over the same document. -fn upsert_payload_id(txn_json: &JsonValue, named_graphs: &[NamedGraphBlock]) -> u64 { +/// itself. +fn upsert_payload_id(txn_json: &JsonValue) -> u64 { use std::io::Write; use xxhash_rust::xxh64::Xxh64; @@ -64,16 +61,6 @@ fn upsert_payload_id(txn_json: &JsonValue, named_graphs: &[NamedGraphBlock]) -> let mut w = HashWriter(Xxh64::new(0)); let _ = w.write_all(b"fluree:upsert\0"); let _ = serde_json::to_writer(&mut w, txn_json); - for block in named_graphs { - let _ = w.write_all(b"\0graph\0"); - let _ = w.write_all(block.iri.as_bytes()); - for triple in &block.triples { - let _ = write!(w, "\0{triple:?}"); - } - for reified in &block.reified { - let _ = write!(w, "\0{reified:?}"); - } - } w.0.digest() } @@ -284,9 +271,10 @@ impl SequentialStager { /// the fact, or retracted then re-asserted when it has it. Last-wins /// alone would commit the final op — a retraction of a fact the ledger /// never held, for `INSERT x ; DELETE x`. Only facts whose first and last - /// ops differ are looked up; the base decides because staging does not - /// filter an assert of a fact already present (or a retract of one - /// absent), so the first op does not say what the base held. + /// ops differ are looked up; the base decides because staging still + /// stages an assert of a fact already present, so a first-op assert does + /// not say what the base held. (A first-op retraction does: staging only + /// retracts a stored fact.) async fn drop_reversed_facts(&mut self) -> Result<()> { let mut keep = vec![true; self.folded.len()]; for (i, flake) in self.folded.iter().enumerate() { @@ -298,6 +286,7 @@ impl SequentialStager { let in_base = crate::rebase::current_asserted_for_key(&self.original, &key) .await? .iter() + .map(fluree_db_transact::StoredFact::flake) .any(|f| f.o == flake.o && f.dt == flake.dt && f.m == flake.m); keep[i] = flake.op != in_base; } @@ -410,6 +399,16 @@ impl<'a> TrackedTransactionInput<'a> { } } +/// A document that does not parse is a Turtle parse error (HTTP 400, +/// `TURTLE_PARSE`), whichever part of it failed; anything else the document +/// raised stays the transaction error it is. +fn rdf_text_error(e: fluree_db_transact::TransactError) -> ApiError { + match e { + fluree_db_transact::TransactError::Turtle(e) => ApiError::Turtle(e), + other => other.into(), + } +} + /// Create a tracker for fuel limits only (no time/policy tracking). /// /// This mirrors query behavior: even non-tracked transactions respect max-fuel. @@ -427,37 +426,6 @@ fn tracker_for_limits(txn_json: &JsonValue) -> Tracker { } } -/// Check if a JSON-LD document represents an empty default graph. -/// -/// This is the case when: -/// - The document is null -/// - The document is an empty array -/// - The document is an object with only JSON-LD keywords (@context, @id, etc.) -/// AND the @graph key is missing or empty -/// -/// This correctly handles envelope-form JSON-LD where data is in @graph. -fn is_empty_default_graph(json: &JsonValue) -> bool { - match json { - JsonValue::Null => true, - JsonValue::Array(arr) => arr.is_empty(), - JsonValue::Object(obj) => { - // Check if there are any non-@ keys (actual data predicates at top level) - let has_data_keys = obj.keys().any(|k| !k.starts_with('@')); - if has_data_keys { - return false; - } - // No data keys at top level, check if @graph has content - match obj.get("@graph") { - Some(JsonValue::Array(arr)) => arr.is_empty(), - Some(JsonValue::Object(inner_obj)) => inner_obj.is_empty(), - Some(_) => false, // @graph has some other non-empty value - None => true, // No @graph key, and no data keys = empty - } - } - _ => false, - } -} - // ============================================================================= // Config-driven transaction helpers (shacl feature only) // ============================================================================= @@ -2351,14 +2319,15 @@ enum GraphOpMode { /// Build the [`Txn`] for a payload scoped to `graph` (see /// [`crate::GraphPayload`]), returning it with the JSON-LD it parsed from, -/// which the staging tail reads limits and the SHACL context from. +/// which the staging tail reads limits and the SHACL context from (`null` +/// for RDF text, which has neither). /// -/// RDF text: a TriG body's blocks are unwrapped in place and the whole body -/// read by the Turtle parser, so block contents get the full Turtle grammar -/// and every format converges on the JSON-LD path. Blocks must name `graph`, -/// and triples outside a block belong to TriG's default graph, so for a named -/// target a body with both is refused rather than guessed at. For the default -/// graph, every block names some other graph. +/// RDF text is parsed once into templates homed on `graph` +/// ([`fluree_db_transact::Placement::Into`]), so block contents get the full +/// Turtle grammar. Blocks must name `graph`, and triples outside a block +/// belong to TriG's default graph, so for a named target a body with both is +/// refused rather than guessed at. For the default graph, every block names +/// some other graph. fn parse_graph_payload<'a>( graph: &GraphSel, payload: crate::GraphPayload<'a>, @@ -2370,39 +2339,29 @@ fn parse_graph_payload<'a>( status: 400, message, }; - let (txn_json, raw_meta) = match payload { - crate::GraphPayload::JsonLd(json) => (std::borrow::Cow::Borrowed(json), None), + let txn_json = match payload { + crate::GraphPayload::JsonLd(json) => std::borrow::Cow::Borrowed(json), crate::GraphPayload::Rdf(text) => { - let trig = fluree_db_transact::unwrap_trig_graph_blocks(text)?; let target = match graph { - GraphSel::Graph(iri) => format!("<{iri}>"), - GraphSel::Default => "the default graph".to_string(), - }; - let named = match graph { GraphSel::Graph(iri) => Some(iri.as_str()), GraphSel::Default => None, }; - if let Some(other) = trig - .graph_iris - .iter() - .find(|iri| Some(iri.as_str()) != named) - { - return Err(bad_request(format!( - "the request targets one graph, {target}; the body also has a GRAPH block \ - for <{other}>" - ))); - } - if trig.mixes_default_and_named { - return Err(bad_request(format!( - "a TriG body holds {target}'s triples either in GRAPH {target} blocks or as \ - default-graph triples, not both" - ))); - } - let nodes = match fluree_graph_turtle::parse_to_json(&trig.turtle)? { - JsonValue::Array(nodes) => nodes, - node => vec![node], - }; - if nodes.is_empty() { + let sync = matches!(mode, GraphOpMode::Sync { .. }); + let (txn, summary) = fluree_db_transact::parse_rdf_text_txn( + text, + TxnType::Insert, + Placement::Into(target), + sync, + txn_opts, + ns_registry, + ) + .map_err(|e| match e { + fluree_db_transact::TransactError::PayloadGraphMismatch(message) => { + bad_request(message) + } + other => rdf_text_error(other), + })?; + if summary.statements == 0 { match mode { GraphOpMode::Sync { allow_empty: true } => {} GraphOpMode::Sync { allow_empty: false } => { @@ -2419,12 +2378,10 @@ fn parse_graph_payload<'a>( } } } - // `"@graph": []` is the JSON-LD explicit-empty form. - let json = serde_json::json!({ "@graph": nodes }); - (std::borrow::Cow::Owned(json), trig.raw_meta) + return Ok((txn, std::borrow::Cow::Owned(JsonValue::Null))); } }; - let mut txn = match mode { + let txn = match mode { GraphOpMode::Sync { .. } => { fluree_db_transact::parse_sync_transaction(&txn_json, graph, txn_opts, ns_registry)? } @@ -2439,226 +2396,9 @@ fn parse_graph_payload<'a>( txn } }; - if let Some(raw_meta) = &raw_meta { - txn.txn_meta - .extend(resolve_trig_meta(raw_meta, ns_registry)?); - } Ok((txn, txn_json)) } -/// Convert named graph blocks to templates in their graphs, returning the -/// templates and the graph IRIs they write to. -fn convert_named_graphs_to_templates( - named_graphs: &[NamedGraphBlock], - ns_registry: &mut NamespaceRegistry, -) -> Result<(Vec, Vec)> { - use fluree_db_transact::{RawObject, RawTerm}; - - let mut templates = Vec::new(); - let mut graph_iris: Vec = Vec::new(); - - // Helper to expand prefixed name to full IRI - fn expand_prefixed_name( - prefix: &str, - local: &str, - prefixes: &rustc_hash::FxHashMap, - ) -> Result { - prefixes - .get(prefix) - .map(|ns| format!("{ns}{local}")) - .ok_or_else(|| { - // Same class as the reserved-predicate refusal below: a - // mistake in a user-authored TriG file, not an engine fault. - // `ApiError::query` rendered it as "Internal error: Query - // error: …", which reads like a bug in Fluree. - ApiError::Transact(fluree_db_transact::TransactError::Parse(format!( - "undefined prefix: {prefix}" - ))) - }) - } - - // Helper to convert RawTerm to TemplateTerm - fn convert_term( - term: &RawTerm, - prefixes: &rustc_hash::FxHashMap, - ns_registry: &mut NamespaceRegistry, - ) -> Result { - match term { - RawTerm::Iri(iri) => { - if let Some(local) = iri.strip_prefix("_:") { - // Stable Fluree blank-node ids address the existing node; - // other labels skolemize fresh at staging. - if let Some(sid) = fluree_db_transact::stable_blank_node_sid_from_label(local) { - return Ok(TemplateTerm::Sid(sid)); - } - Ok(TemplateTerm::BlankNode(local.to_string())) - } else { - Ok(TemplateTerm::Sid(ns_registry.sid_for_iri(iri))) - } - } - RawTerm::PrefixedName { prefix, local } => { - let iri = expand_prefixed_name(prefix, local, prefixes)?; - Ok(TemplateTerm::Sid(ns_registry.sid_for_iri(&iri))) - } - } - } - - // Helper to convert RawObject to TemplateTerm and optional datatype/language - fn convert_object( - obj: &RawObject, - prefixes: &rustc_hash::FxHashMap, - ns_registry: &mut NamespaceRegistry, - ) -> Result<(TemplateTerm, Option)> { - use fluree_db_core::FlakeValue; - match obj { - RawObject::Iri(iri) => { - if let Some(local) = iri.strip_prefix("_:") { - // Stable Fluree blank-node ids resolve to the stored node - // (see convert_term). - if let Some(sid) = fluree_db_transact::stable_blank_node_sid_from_label(local) { - return Ok((TemplateTerm::Sid(sid), None)); - } - Ok((TemplateTerm::BlankNode(local.to_string()), None)) - } else { - Ok((TemplateTerm::Sid(ns_registry.sid_for_iri(iri)), None)) - } - } - RawObject::PrefixedName { prefix, local } => { - let iri = expand_prefixed_name(prefix, local, prefixes)?; - Ok((TemplateTerm::Sid(ns_registry.sid_for_iri(&iri)), None)) - } - RawObject::String(s) => Ok((TemplateTerm::Value(FlakeValue::String(s.clone())), None)), - RawObject::Integer(n) => Ok((TemplateTerm::Value(FlakeValue::Long(*n)), None)), - RawObject::Double(n) => Ok((TemplateTerm::Value(FlakeValue::Double(*n)), None)), - RawObject::Boolean(b) => Ok((TemplateTerm::Value(FlakeValue::Boolean(*b)), None)), - RawObject::LangString { value, lang } => Ok(( - TemplateTerm::Value(FlakeValue::String(value.clone())), - Some(DatatypeConstraint::LangTag(Arc::from(lang.as_str()))), - )), - RawObject::TypedLiteral { value, datatype } => { - // Parse the lexical with the shared coercion (exact decimals, - // integers, temporals, ...) — storing the raw String here - // would persist a string flake mislabeled with the datatype, - // diverging from the bulk-import path. Unknown datatypes keep - // the lexical as a string, matching import behavior. - let dt_sid = ns_registry.sid_for_iri(datatype); - let fv = fluree_db_core::coerce::coerce_string_value(value, datatype) - .unwrap_or_else(|_| FlakeValue::String(value.clone())); - Ok(( - TemplateTerm::Value(fv), - Some(DatatypeConstraint::Explicit(dt_sid)), - )) - } - } - } - - for block in named_graphs { - graph_iris.push(block.iri.clone()); - let graph: std::sync::Arc = std::sync::Arc::from(block.iri.as_str()); - - // Convert each triple in this graph block - for triple in &block.triples { - let subject = triple.subject.as_ref().ok_or_else(|| { - ApiError::Transact(fluree_db_transact::TransactError::Parse( - "named graph triple missing subject".to_string(), - )) - })?; - let subject_term = convert_term(subject, &block.prefixes, ns_registry)?; - let predicate_term = convert_term(&triple.predicate, &block.prefixes, ns_registry)?; - - // Reserved-predicate firewall. The JSON-LD, SPARQL UPDATE and - // Turtle surfaces all refuse a hand-written `f:reifies*` - // statement, because an attachment bundle is only well-formed if - // the annotation syntax built it. TriG `GRAPH { … }` blocks come - // through here instead of `FlakeSink::build_flake`, so they had no - // check at all and such a triple landed. - if let TemplateTerm::Sid(p) = &predicate_term { - if fluree_db_core::is_reserved_reifies_predicate(p) { - let iri = ns_registry.get_prefix(p.namespace_code).map_or_else( - || p.name.to_string(), - |prefix| format!("{prefix}{}", p.name), - ); - // A transact error, not a query one: this is user-authored - // input being refused at write time, and `ApiError::query` - // rendered it as "Internal error: Query error: …", which - // reads like a bug in the engine rather than a problem with - // the statement. `ApiError::Transact(_)` maps to - // `errors::INVALID_TRANSACTION` / HTTP 422 - // (`fluree-db-server/src/error.rs`). - return Err(ApiError::Transact( - fluree_db_transact::TransactError::UnsupportedFeature(format!( - "'{iri}' is a system-controlled predicate; use the RDF 1.2 \ - annotation syntax (`~ {{| ... |}}` or \ - `<< s p o >>`) instead of writing f:reifies* triples by hand" - )), - )); - } - } - - for obj in &triple.objects { - let (object_term, dtc) = convert_object(obj, &block.prefixes, ns_registry)?; - let mut template = - TripleTemplate::new(subject_term.clone(), predicate_term.clone(), object_term); - template = template.in_graph(std::sync::Arc::clone(&graph)); - if let Some(dtc) = dtc { - template = template.with_dtc(dtc); - } - templates.push(template); - } - } - - // TriG-star: one `f:reifies*` bundle per reifier attachment, in the - // same graph as the edge it reifies — the shape the JSON-LD - // `@annotation` sibling produces (f:reifiesGraph present, no - // f:reifiesDatatype, f:reifiesLang for language-tagged objects). - if !block.reified.is_empty() { - use fluree_db_core::namespaces::{ - reifies_graph_sid, reifies_lang_sid, reifies_object_sid, reifies_predicate_sid, - reifies_subject_sid, - }; - let graph_sid = ns_registry.sid_for_iri(&block.iri); - for r in &block.reified { - let ann = convert_term(&r.reifier, &block.prefixes, ns_registry)?; - let s = convert_term(&r.subject, &block.prefixes, ns_registry)?; - let p = convert_term(&r.predicate, &block.prefixes, ns_registry)?; - let (o, dtc) = convert_object(&r.object, &block.prefixes, ns_registry)?; - let lang = match &dtc { - Some(DatatypeConstraint::LangTag(lang)) => Some(lang.to_string()), - _ => None, - }; - let mut push = |pred: &fluree_db_core::Sid, - obj: TemplateTerm, - dtc: Option| { - let mut t = - TripleTemplate::new(ann.clone(), TemplateTerm::Sid(pred.clone()), obj) - .in_graph(std::sync::Arc::clone(&graph)); - if let Some(d) = dtc { - t = t.with_dtc(d); - } - templates.push(t); - }; - push( - reifies_graph_sid(), - TemplateTerm::Sid(graph_sid.clone()), - None, - ); - push(reifies_subject_sid(), s, None); - push(reifies_predicate_sid(), p, None); - if let Some(lang) = lang { - push( - reifies_lang_sid(), - TemplateTerm::Value(fluree_db_core::FlakeValue::String(lang)), - None, - ); - } - push(reifies_object_sid(), o, dtc); - } - } - } - - Ok((templates, graph_iris)) -} - /// Stage options for a write: backpressure against `index_config`, modify /// policy, and fuel tracking when the tracker is enabled. fn stage_options<'a>( @@ -2771,68 +2511,12 @@ impl crate::Fluree { txn_opts: TxnOpts, index_config: Option<&IndexConfig>, ) -> Result { - self.stage_transaction_with_trig_meta( - ledger, - txn_type, - txn_json, - txn_opts, - index_config, - None, - ) - .await - } - - /// Stage a transaction with optional TriG transaction metadata. - /// - /// This is the internal implementation that handles both JSON-LD and TriG inputs. - /// For TriG inputs, the `trig_meta` parameter contains pre-parsed metadata that - /// will be resolved and merged into the transaction's txn_meta. - pub async fn stage_transaction_with_trig_meta( - &self, - ledger: LedgerState, - txn_type: TxnType, - txn_json: &JsonValue, - txn_opts: TxnOpts, - index_config: Option<&IndexConfig>, - trig_meta: Option<&RawTrigMeta>, - ) -> Result { - self.stage_transaction_with_named_graphs( - ledger, - txn_type, - txn_json, - txn_opts, - index_config, - trig_meta, - &[], - ) - .await - } - - /// Stage a transaction with optional TriG transaction metadata and named graphs. - /// - /// This is the full implementation that handles: - /// - JSON-LD transactions (default graph) - /// - TriG txn-meta (commit metadata) - /// - TriG named graphs (user-defined graphs with separate g_id) - #[allow(clippy::too_many_arguments)] - pub async fn stage_transaction_with_named_graphs( - &self, - ledger: LedgerState, - txn_type: TxnType, - txn_json: &JsonValue, - txn_opts: TxnOpts, - index_config: Option<&IndexConfig>, - trig_meta: Option<&RawTrigMeta>, - named_graphs: &[NamedGraphBlock], - ) -> Result { - self.stage_transaction_with_named_graphs_tracked( + self.stage_transaction_tracked( ledger, txn_type, txn_json, txn_opts, index_config, - trig_meta, - named_graphs, None, None, ) @@ -2889,23 +2573,22 @@ impl crate::Fluree { } } - /// Stage a transaction with optional TriG metadata, named graphs, external tracker, - /// and policy context. + /// Stage a JSON-LD transaction with an optional external tracker and + /// policy context. /// /// When `external_tracker` is provided, it is used for fuel accounting instead of /// the default limits-only tracker derived from the transaction body opts. /// When `policy` is provided, modify policies are enforced on staged flakes - /// (unless the wrapper is root). + /// (unless the wrapper is root). RDF text is staged by its own lane + /// (`stage_rdf_text_tracked`), never as JSON-LD. #[allow(clippy::too_many_arguments)] - pub async fn stage_transaction_with_named_graphs_tracked( + pub async fn stage_transaction_tracked( &self, ledger: LedgerState, txn_type: TxnType, txn_json: &JsonValue, txn_opts: TxnOpts, index_config: Option<&IndexConfig>, - trig_meta: Option<&RawTrigMeta>, - named_graphs: &[NamedGraphBlock], external_tracker: Option<&Tracker>, policy: Option<&crate::PolicyContext>, ) -> Result { @@ -2929,40 +2612,16 @@ impl crate::Fluree { // id still wins. let mut txn_opts = txn_opts; if txn_type == TxnType::Upsert && txn_opts.skolem_txn_id.is_none() { - let scope = - fluree_db_core::skolem::doc_scope(upsert_payload_id(txn_json, named_graphs)); + let scope = fluree_db_core::skolem::doc_scope(upsert_payload_id(txn_json)); txn_opts.skolem_txn_id = Some(format!("upsert{scope}")); } - // Handle case where default graph is empty but named graphs are present - // (e.g., TriG with only GRAPH blocks and no default graph triples) - let mut txn = if is_empty_default_graph(txn_json) && !named_graphs.is_empty() { - // Create empty transaction of the appropriate type - match txn_type { - TxnType::Insert => Txn::insert().with_opts(txn_opts), - TxnType::Upsert => Txn::upsert().with_opts(txn_opts), - TxnType::Update => Txn::update().with_opts(txn_opts), - } - } else { + let txn = { let parse_span = tracing::debug_span!("txn_parse", txn_type = ?txn_type); let _guard = parse_span.enter(); parse_transaction(txn_json, txn_type, txn_opts, &mut ns_registry)? }; - // If TriG metadata was extracted, resolve it and merge into txn_meta - if let Some(raw_meta) = trig_meta { - let resolved = resolve_trig_meta(raw_meta, &mut ns_registry)?; - txn.txn_meta.extend(resolved); - } - - // Convert named graph blocks to TripleTemplates and merge into the transaction - if !named_graphs.is_empty() { - let (named_graph_templates, named_graph_iris) = - convert_named_graphs_to_templates(named_graphs, &mut ns_registry)?; - txn.insert_templates.extend(named_graph_templates); - txn.write_graphs.extend(named_graph_iris); - } - self.stage_built_txn_tracked( ledger, txn, @@ -3087,6 +2746,55 @@ impl crate::Fluree { .await } + /// Stage RDF text (Turtle or TriG) as a `txn_type` transaction. + /// + /// The text is parsed once, by the Turtle parser, into templates + /// ([`fluree_db_transact::parse_rdf_text_txn`]); nothing is re-encoded + /// as JSON-LD. Statements land where the document puts them: the default + /// graph's in the default graph, each block's in its graph, and a + /// `<#txn-meta>` block's in the commit metadata. An upsert scopes its + /// blank nodes to the document's content, so the same document upserted + /// again addresses the same nodes. + #[allow(clippy::too_many_arguments)] + pub(crate) async fn stage_rdf_text_tracked( + &self, + ledger: LedgerState, + txn_type: TxnType, + text: &str, + txn_opts: TxnOpts, + index_config: Option<&IndexConfig>, + external_tracker: Option<&Tracker>, + policy: Option<&crate::PolicyContext>, + ) -> Result { + let mut ns_registry = NamespaceRegistry::from_db(&ledger.snapshot); + let (txn, _) = { + let parse_span = + tracing::debug_span!("txn_parse", txn_type = ?txn_type, rdf_bytes = text.len()); + let _guard = parse_span.enter(); + fluree_db_transact::parse_rdf_text_txn( + text, + txn_type, + Placement::AsWritten, + false, + txn_opts, + &mut ns_registry, + ) + .map_err(rdf_text_error)? + }; + // RDF text has no JSON body: no `opts.maxFuel` to read and no + // `@context` to compact violation messages with. + self.stage_built_txn_tracked( + ledger, + txn, + ns_registry, + &JsonValue::Null, + index_config, + external_tracker, + policy, + ) + .await + } + /// Shared staging tail for a fully-built JSON-LD [`Txn`]: uniqueness / /// SHACL / reasoning validation and [`StageResult`] assembly. #[allow(clippy::too_many_arguments)] @@ -3680,88 +3388,6 @@ impl crate::Fluree { .await } - /// Execute a transaction with optional TriG metadata. - /// - /// This is similar to `transact` but accepts pre-extracted TriG metadata - /// from Turtle inputs that had GRAPH blocks. - #[allow(clippy::too_many_arguments)] - pub async fn transact_with_trig_meta( - &self, - ledger: LedgerState, - txn_type: TxnType, - txn_json: &JsonValue, - txn_opts: TxnOpts, - commit_opts: CommitOpts, - index_config: &IndexConfig, - trig_meta: Option<&RawTrigMeta>, - ) -> Result { - let store_raw_txn = txn_opts.store_raw_txn.unwrap_or(false); - - // Spawn raw_txn upload in parallel with staging when opted in. - let commit_opts = if commit_opts.raw_txn_upload.is_none() && store_raw_txn { - let content_store = self.content_store(ledger.ledger_id()); - commit_opts.with_raw_txn_spawned(content_store, txn_json.clone()) - } else { - commit_opts - }; - - let staged = self - .stage_transaction_with_trig_meta( - ledger, - txn_type, - txn_json, - txn_opts, - Some(index_config), - trig_meta, - ) - .await?; - self.commit_stage_result(staged, txn_type, commit_opts, index_config) - .await - } - - /// Execute a transaction with optional TriG metadata and named graphs. - /// - /// This is the full implementation that handles: - /// - JSON-LD transactions (default graph) - /// - TriG txn-meta (commit metadata) - /// - TriG named graphs (user-defined graphs with separate g_id) - #[allow(clippy::too_many_arguments)] - pub async fn transact_with_named_graphs( - &self, - ledger: LedgerState, - txn_type: TxnType, - txn_json: &JsonValue, - txn_opts: TxnOpts, - commit_opts: CommitOpts, - index_config: &IndexConfig, - trig_meta: Option<&RawTrigMeta>, - named_graphs: &[NamedGraphBlock], - ) -> Result { - let store_raw_txn = txn_opts.store_raw_txn.unwrap_or(false); - - // Spawn raw_txn upload in parallel with staging when opted in. - let commit_opts = if commit_opts.raw_txn_upload.is_none() && store_raw_txn { - let content_store = self.content_store(ledger.ledger_id()); - commit_opts.with_raw_txn_spawned(content_store, txn_json.clone()) - } else { - commit_opts - }; - - let staged = self - .stage_transaction_with_named_graphs( - ledger, - txn_type, - txn_json, - txn_opts, - Some(index_config), - trig_meta, - named_graphs, - ) - .await?; - self.commit_stage_result(staged, txn_type, commit_opts, index_config) - .await - } - /// Insert new data into the ledger /// /// Fails if any subject with a concrete `@id` already has triples in the ledger. @@ -3868,8 +3494,8 @@ impl crate::Fluree { /// Stage a Turtle or TriG INSERT. /// /// Turtle is parsed directly to flakes by `FlakeSink`, bypassing JSON-LD / - /// IR. A TriG document (graph blocks, `<#txn-meta>`) is staged through the - /// named-graph path TriG upsert uses, with insert semantics. + /// IR. A TriG document (graph blocks, `<#txn-meta>`) is parsed into + /// templates, as a TriG upsert is, and staged with insert semantics. pub async fn stage_turtle_insert( &self, ledger: LedgerState, @@ -4052,9 +3678,10 @@ impl crate::Fluree { }) } - /// Stage a document the streaming Turtle parser rejected as a TriG insert, - /// through the same named-graph path TriG upsert takes. A document with no - /// graph blocks and no txn-meta isn't TriG, so its Turtle error stands. + /// Stage a document the streaming Turtle parser rejected as a TriG insert: + /// parsed once into templates, as a TriG upsert is, with insert semantics. + /// A document with no graph block (`<#txn-meta>` included) isn't TriG, so + /// its Turtle error stands; a malformed block is reported as one. #[allow(clippy::too_many_arguments)] async fn stage_trig_insert( &self, @@ -4066,28 +3693,30 @@ impl crate::Fluree { tracker: Option<&Tracker>, policy: Option<&crate::PolicyContext>, ) -> Result { - // Most rejected documents are plain Turtle with a typo; rule TriG out - // without copying them. - if !fluree_db_transact::might_contain_graph_block(trig) { - return Err(turtle_err); - } - let phase1 = fluree_db_transact::parse_trig_phase1(trig)?; - if phase1.named_graphs.is_empty() && phase1.raw_meta.is_none() { + // Most rejected documents are plain Turtle with a typo; the locator + // rules TriG out from tokens alone, and a TriG document is located + // once. + let mut ns_registry = NamespaceRegistry::from_db(&ledger.snapshot); + let parsed = { + let parse_span = + tracing::debug_span!("txn_parse", txn_type = "insert", rdf_bytes = trig.len()); + let _guard = parse_span.enter(); + fluree_db_transact::parse_trig_txn(trig, TxnType::Insert, txn_opts, &mut ns_registry) + .map_err(rdf_text_error)? + }; + let Some((txn, _)) = parsed else { return Err(turtle_err); - } + }; stamp_fast_path( TURTLE_INSERT_SITE, FastPathOutcome::Fallback(FastPathFallback::GateDeclined), ); - let txn_json = fluree_graph_turtle::parse_to_json(&phase1.turtle)?; - self.stage_transaction_with_named_graphs_tracked( + self.stage_built_txn_tracked( ledger, - TxnType::Insert, - &txn_json, - txn_opts, + txn, + ns_registry, + &JsonValue::Null, index_config, - phase1.raw_meta.as_ref(), - &phase1.named_graphs, tracker, policy, ) @@ -4152,16 +3781,18 @@ impl crate::Fluree { .await } - /// Upsert data from Turtle format + /// Upsert data from Turtle or TriG /// - /// Parses the Turtle input and upserts it into the ledger. - /// For each (subject, predicate) pair, existing values are retracted - /// before new values are asserted. + /// Parses the document once and upserts it into the ledger: for each + /// (graph, subject, predicate) it names, the current values are + /// retracted and the document's values asserted. A TriG document's + /// blocks upsert into their graphs, and a `<#txn-meta>` block becomes + /// commit metadata. /// /// # Arguments /// /// * `ledger` - The ledger state (consumed) - /// * `turtle` - Turtle (TTL) format data + /// * `turtle` - Turtle or TriG text /// /// # Example /// @@ -4172,11 +3803,18 @@ impl crate::Fluree { /// "#).await?; /// ``` pub async fn upsert_turtle(&self, ledger: LedgerState, turtle: &str) -> Result { - let data = fluree_graph_turtle::parse_to_json(turtle)?; - self.upsert(ledger, &data).await + let index_config = self.default_index_config(); + self.upsert_turtle_with_opts( + ledger, + turtle, + TxnOpts::default(), + CommitOpts::default(), + &index_config, + ) + .await } - /// Upsert data from Turtle format with options + /// Upsert data from Turtle or TriG with options /// /// Same as `upsert_turtle` but allows custom transaction and commit options. /// Prefer using the builder API: `fluree.transact(ledger).upsert_turtle(ttl).txn_opts(...).execute()`. @@ -4189,8 +3827,28 @@ impl crate::Fluree { commit_opts: CommitOpts, index_config: &IndexConfig, ) -> Result { - let data = fluree_graph_turtle::parse_to_json(turtle)?; - self.upsert_with_opts(ledger, &data, txn_opts, commit_opts, index_config) + let store_raw_txn = txn_opts.store_raw_txn.unwrap_or(false); + + // The raw transaction is the text, as for a Turtle insert. + let commit_opts = if commit_opts.raw_txn_upload.is_none() && store_raw_txn { + let content_store = self.content_store(ledger.ledger_id()); + commit_opts.with_raw_txn_spawned(content_store, JsonValue::String(turtle.to_string())) + } else { + commit_opts + }; + + let staged = self + .stage_rdf_text_tracked( + ledger, + TxnType::Upsert, + turtle, + txn_opts, + Some(index_config), + None, + None, + ) + .await?; + self.commit_stage_result(staged, TxnType::Upsert, commit_opts, index_config) .await } @@ -4450,7 +4108,6 @@ fn _ensure_error_used(e: ApiError) -> ApiError { #[cfg(test)] mod tests { use super::*; - use fluree_db_transact::{RawObject, RawTerm, RawTriple}; /// A write rejected at max novelty must ask the indexer for a build. It is /// the one condition no post-commit trigger can cover, because there is no @@ -4590,137 +4247,4 @@ mod tests { without it this write family never asks for the thing that unblocks it" ); } - - /// A hand-written `f:reifies*` triple inside a TriG `GRAPH` block is - /// refused, the way it is on every other write surface. - /// - /// This path does not go through `FlakeSink::build_flake`, which is where - /// the Turtle firewall lives, so without its own check the statement - /// landed and produced an attachment bundle no annotation syntax built. - #[test] - fn named_graph_block_refuses_a_reserved_reifies_predicate() { - let block = NamedGraphBlock { - iri: "http://example.org/g1".to_string(), - triples: vec![RawTriple { - subject: Some(RawTerm::Iri("http://example.org/claim1".to_string())), - predicate: RawTerm::Iri(fluree_vocab::reifies_iris::SUBJECT.to_string()), - objects: vec![RawObject::Iri("http://example.org/evil".to_string())], - }], - reified: Vec::new(), - prefixes: Default::default(), - }; - let mut ns = NamespaceRegistry::new(); - let err = convert_named_graphs_to_templates(&[block], &mut ns) - .expect_err("a reserved predicate in a GRAPH block must be refused"); - let msg = err.to_string(); - assert!( - msg.contains("system-controlled predicate"), - "unexpected error: {msg}" - ); - assert_transact_not_query(&err, "reserved f:reifies* predicate"); - } - - /// Every refusal in `convert_named_graphs_to_templates` is a transact - /// error, not a query one. - /// - /// All of them reject user-authored TriG at write time, but through - /// `ApiError::query` they arrived as `Internal error: Query error: …`, - /// which reads like a fault in the engine rather than a mistake in the - /// file — and maps to the wrong HTTP class, since `ApiError::Transact(_)` - /// is what `fluree-db-server/src/error.rs` maps to - /// `errors::INVALID_TRANSACTION` / 422. - /// - /// Asserted on the variant rather than the message, because the message is - /// identical either way: a text assertion passes whichever type comes back - /// and so pins nothing. - fn assert_transact_not_query(err: &ApiError, label: &str) { - assert!( - matches!(err, ApiError::Transact(_)), - "{label} must surface as ApiError::Transact, got: {err:?}" - ); - let rendered = err.to_string(); - assert!( - !rendered.starts_with("Internal error: Query error:"), - "{label} must not render as an engine fault; got: {rendered}" - ); - } - - #[test] - fn an_undefined_prefix_in_a_graph_block_is_a_transact_error() { - let block = NamedGraphBlock { - iri: "http://example.org/g1".to_string(), - triples: vec![RawTriple { - subject: Some(RawTerm::PrefixedName { - prefix: "nope".to_string(), - local: "a".to_string(), - }), - predicate: RawTerm::Iri("http://example.org/p".to_string()), - objects: vec![RawObject::Iri("http://example.org/b".to_string())], - }], - reified: Vec::new(), - prefixes: Default::default(), - }; - let mut ns = NamespaceRegistry::new(); - let err = convert_named_graphs_to_templates(&[block], &mut ns) - .expect_err("an undefined prefix must be refused"); - assert!( - err.to_string().contains("undefined prefix"), - "unexpected error: {err}" - ); - assert_transact_not_query(&err, "undefined prefix"); - } - - #[test] - fn a_graph_block_triple_without_a_subject_is_a_transact_error() { - let block = NamedGraphBlock { - iri: "http://example.org/g1".to_string(), - triples: vec![RawTriple { - subject: None, - predicate: RawTerm::Iri("http://example.org/p".to_string()), - objects: vec![RawObject::Iri("http://example.org/b".to_string())], - }], - reified: Vec::new(), - prefixes: Default::default(), - }; - let mut ns = NamespaceRegistry::new(); - let err = convert_named_graphs_to_templates(&[block], &mut ns) - .expect_err("a subjectless triple must be refused"); - assert_transact_not_query(&err, "named graph triple missing subject"); - } - - /// TriG named-graph blocks (upsert/insert-turtle path): a stable - /// `_:fdb-...` id must resolve to the stored node's Sid, while ordinary - /// labels stay `TemplateTerm::BlankNode` for fresh skolemization at - /// staging. - #[test] - fn named_graph_stable_blank_node_resolves_to_sid() { - let block = NamedGraphBlock { - iri: "http://example.org/g1".to_string(), - triples: vec![RawTriple { - subject: Some(RawTerm::Iri("_:fdb-1234-0-b0".to_string())), - predicate: RawTerm::Iri("http://example.org/knows".to_string()), - objects: vec![RawObject::Iri("_:other".to_string())], - }], - reified: Vec::new(), - prefixes: Default::default(), - }; - let mut ns = NamespaceRegistry::new(); - let (templates, _delta) = - convert_named_graphs_to_templates(&[block], &mut ns).expect("convert"); - assert_eq!(templates.len(), 1); - - let expected = ns.blank_node_sid("1234-0-b0"); - match &templates[0].subject { - TemplateTerm::Sid(sid) => { - assert_eq!(sid, &expected, "stable id must address the stored node"); - } - other => panic!("stable id must resolve to a Sid, got {other:?}"), - } - match &templates[0].object { - TemplateTerm::BlankNode(label) => { - assert_eq!(label, "other", "ordinary labels keep fresh-mint semantics"); - } - other => panic!("ordinary label must stay BlankNode, got {other:?}"), - } - } } diff --git a/fluree-db-api/src/tx_builder.rs b/fluree-db-api/src/tx_builder.rs index 071fd20d83..84b67a9c5f 100644 --- a/fluree-db-api/src/tx_builder.rs +++ b/fluree-db-api/src/tx_builder.rs @@ -33,8 +33,8 @@ use fluree_db_nameservice::NsRecord; use fluree_db_novelty::Novelty; use fluree_db_transact::GraphSel; use fluree_db_transact::{ - lower_sparql_update_request, parse_trig_phase1, CommitOpts, NamedGraphBlock, NamespaceRegistry, - RawTrigMeta, TransactError, Txn, TxnOpts, TxnType, + lower_sparql_update_request, CommitOpts, NamespaceRegistry, TransactError, Txn, TxnOpts, + TxnType, }; use rustc_hash::FxHashSet; use std::collections::HashMap; @@ -410,14 +410,6 @@ impl GraphOp<'_> { } } -/// Result of parsing a transaction operation to JSON. -/// For Turtle inputs with TriG GRAPH blocks, also includes raw txn-meta and named graphs. -pub(crate) struct ParsedOperation { - pub json: JsonValue, - pub trig_meta: Option, - pub named_graphs: Vec, -} - impl TransactOperation<'_> { /// Get the `TxnType` for this operation. pub(crate) fn txn_type(&self) -> TxnType { @@ -431,51 +423,25 @@ impl TransactOperation<'_> { } } - /// Parse the operation to JSON, extracting TriG txn-meta and named graphs from Turtle inputs. - /// - /// For Turtle inputs with `GRAPH <...> { ... }` blocks, this extracts: - /// - Metadata from txn-meta graph - /// - Named graph blocks for user-defined graphs - /// - /// The metadata can be resolved to `TxnMetaEntry` using `resolve_trig_meta()` once a - /// `NamespaceRegistry` is available. Named graphs are converted to - /// `TripleTemplate`s with appropriate graph_id during staging. - pub(crate) fn to_json_with_trig_meta(&self) -> Result { + /// The operation's JSON-LD body. RDF text and graph operations are + /// staged by their own lanes and never read as JSON-LD. + pub(crate) fn to_json(&self) -> Result { match self { - TransactOperation::InsertJson(j) => Ok(ParsedOperation { - json: (*j).clone(), - trig_meta: None, - named_graphs: Vec::new(), - }), - TransactOperation::UpsertJson(j) => Ok(ParsedOperation { - json: (*j).clone(), - trig_meta: None, - named_graphs: Vec::new(), - }), - TransactOperation::UpdateJson(j) => Ok(ParsedOperation { - json: (*j).clone(), - trig_meta: None, - named_graphs: Vec::new(), - }), + TransactOperation::InsertJson(j) + | TransactOperation::UpsertJson(j) + | TransactOperation::UpdateJson(j) => Ok((*j).clone()), // Every staging path dispatches graph ops first; parsing one as a // plain insert would drop its graph scope (and a sync's // retraction wave). TransactOperation::Graph(_) => Err(ApiError::internal( "graph operations must be staged through stage_graph_op_tracked", )), - TransactOperation::InsertTurtle(ttl) | TransactOperation::UpsertTurtle(ttl) => { - // Phase 1: Extract TriG GRAPH block (if present) - let phase1 = parse_trig_phase1(ttl)?; - - // Parse cleaned Turtle to JSON - let json = fluree_graph_turtle::parse_to_json(&phase1.turtle)?; - - Ok(ParsedOperation { - json, - trig_meta: phase1.raw_meta, - named_graphs: phase1.named_graphs, - }) - } + // RDF text is parsed once, into templates, by the staging path + // (`stage_turtle_insert_with_opts`, `stage_rdf_text_tracked`); + // reading it here would re-encode it as JSON-LD. + TransactOperation::InsertTurtle(_) | TransactOperation::UpsertTurtle(_) => Err( + ApiError::internal("RDF text must be staged through its own lane, not as JSON-LD"), + ), } } } @@ -1203,6 +1169,13 @@ fn remap_sid(sid: &mut Sid, remap: &HashMap) { pub(crate) enum OpPlan<'a> { InsertTurtle(&'a str), + /// RDF text staged as templates (an upsert), parsed against whichever + /// state it is staged on: parsing allocates namespace codes relative to + /// that state's table. + Rdf { + txn_type: TxnType, + text: &'a str, + }, /// A SPARQL UPDATE request, parsed and lowered against whichever state /// it is staged on — lowering allocates namespace codes relative to /// that state's table. @@ -1210,28 +1183,26 @@ pub(crate) enum OpPlan<'a> { JsonLike { txn_type: TxnType, txn_json: JsonValue, - trig_meta: Option, - named_graphs: Vec, }, /// A write scoped to one graph (see [`GraphOp`]). Graph(GraphOp<'a>), } impl<'a> OpPlan<'a> { - /// Pre-parse a [`TransactOperation`] into an [`OpPlan`], extracting TriG - /// metadata and named graphs for Turtle inputs. + /// Pre-parse a [`TransactOperation`] into an [`OpPlan`]. pub(crate) fn from_op(op: TransactOperation<'a>) -> Result { match op { TransactOperation::InsertTurtle(turtle) => Ok(OpPlan::InsertTurtle(turtle)), + TransactOperation::UpsertTurtle(text) => Ok(OpPlan::Rdf { + txn_type: TxnType::Upsert, + text, + }), TransactOperation::Graph(op) => Ok(OpPlan::Graph(op)), _ => { let txn_type = op.txn_type(); - let parsed = op.to_json_with_trig_meta()?; Ok(OpPlan::JsonLike { txn_type, - txn_json: parsed.json, - trig_meta: parsed.trig_meta, - named_graphs: parsed.named_graphs, + txn_json: op.to_json()?, }) } } @@ -1504,6 +1475,28 @@ impl Fluree { return Ok((stage_result, TxnType::Insert, commit_opts, None)); } + // RDF text upsert: parsed once into templates. + if let TransactOperation::UpsertTurtle(text) = op { + let commit_opts = self.maybe_spawn_txn_upload( + core.commit_opts, + &ledger_id, + serde_json::Value::String(text.to_string()), + store_raw_txn, + ); + let stage_result = self + .stage_rdf_text_tracked( + ledger_state, + TxnType::Upsert, + text, + core.txn_opts, + Some(index_config), + tracker_ref, + core.policy.as_ref(), + ) + .await?; + return Ok((stage_result, TxnType::Upsert, commit_opts, None)); + } + // Graph ops: dedicated staging (graph scope, sync retraction wave). // Must be dispatched before the JSON-like fallthrough, which would // otherwise stage the payload as a plain default-graph insert. @@ -1527,12 +1520,9 @@ impl Fluree { return Ok((stage_result, op.txn_type(), commit_opts, None)); } - // JSON-like operation: parse, extracting TriG metadata + named graphs. + // JSON-LD operation. let txn_type = op.txn_type(); - let parsed = op.to_json_with_trig_meta()?; - let txn_json = parsed.json; - let trig_meta = parsed.trig_meta; - let named_graphs = parsed.named_graphs; + let txn_json = op.to_json()?; let commit_opts = self.maybe_spawn_txn_upload( core.commit_opts, @@ -1541,14 +1531,12 @@ impl Fluree { store_raw_txn, ); let stage_result = self - .stage_transaction_with_named_graphs_tracked( + .stage_transaction_tracked( ledger_state, txn_type, &txn_json, core.txn_opts, Some(index_config), - trig_meta.as_ref(), - &named_graphs, tracker_ref, core.policy.as_ref(), ) @@ -1616,12 +1604,27 @@ impl Fluree { .await?; Ok((stage_result, TxnType::Insert, commit_opts)) } - OpPlan::JsonLike { - txn_type, - txn_json, - trig_meta, - named_graphs, - } => { + OpPlan::Rdf { txn_type, text } => { + let commit_opts = self.maybe_spawn_txn_upload( + commit_opts_base.clone(), + &ledger_id, + serde_json::Value::String((*text).to_string()), + store_raw_txn, + ); + let stage_result = self + .stage_rdf_text_tracked( + ledger_state, + *txn_type, + text, + txn_opts, + Some(index_config), + tracker_ref, + None, + ) + .await?; + Ok((stage_result, *txn_type, commit_opts)) + } + OpPlan::JsonLike { txn_type, txn_json } => { let commit_opts = self.maybe_spawn_txn_upload( commit_opts_base.clone(), &ledger_id, @@ -1629,14 +1632,12 @@ impl Fluree { store_raw_txn, ); let stage_result = self - .stage_transaction_with_named_graphs_tracked( + .stage_transaction_tracked( ledger_state, *txn_type, txn_json, txn_opts, Some(index_config), - trig_meta.as_ref(), - named_graphs, tracker_ref, policy, ) diff --git a/fluree-db-api/tests/grp_ledger.rs b/fluree-db-api/tests/grp_ledger.rs index c8ce45dba3..4c96259170 100644 --- a/fluree-db-api/tests/grp_ledger.rs +++ b/fluree-db-api/tests/grp_ledger.rs @@ -26,6 +26,8 @@ mod it_ledger_lifecycle; mod it_merge; #[path = "it_merge_preview.rs"] mod it_merge_preview; +#[path = "it_named_graph_conflict_retractions.rs"] +mod it_named_graph_conflict_retractions; #[path = "it_profile_ledger.rs"] mod it_profile_ledger; #[path = "it_push_after_merge.rs"] @@ -38,6 +40,8 @@ mod it_rebase; mod it_refresh; #[path = "it_revert.rs"] mod it_revert; +#[path = "it_revert_legacy_phantom.rs"] +mod it_revert_legacy_phantom; #[path = "it_revert_preview.rs"] mod it_revert_preview; #[path = "it_stable_hashes.rs"] diff --git a/fluree-db-api/tests/grp_misc.rs b/fluree-db-api/tests/grp_misc.rs index 7ae4cf774a..3ae987191d 100644 --- a/fluree-db-api/tests/grp_misc.rs +++ b/fluree-db-api/tests/grp_misc.rs @@ -1,6 +1,8 @@ #[path = "support/mod.rs"] mod support; +#[path = "it_annotation_delete_named_graph.rs"] +mod it_annotation_delete_named_graph; #[path = "it_annotation_filter_pushdown.rs"] mod it_annotation_filter_pushdown; #[path = "it_compile_breakdown.rs"] diff --git a/fluree-db-api/tests/grp_transact.rs b/fluree-db-api/tests/grp_transact.rs index 7dcb837b3a..f28b447827 100644 --- a/fluree-db-api/tests/grp_transact.rs +++ b/fluree-db-api/tests/grp_transact.rs @@ -11,12 +11,16 @@ mod it_concurrent_update_reconcile; mod it_cypher_txn; #[path = "it_default_graph_name.rs"] mod it_default_graph_name; +#[path = "it_delete_stored_facts.rs"] +mod it_delete_stored_facts; #[path = "it_enforce_unique_upsert_indexed.rs"] mod it_enforce_unique_upsert_indexed; #[path = "it_jsonld_empty_list.rs"] mod it_jsonld_empty_list; #[path = "it_raw_txn_parallel_upload.rs"] mod it_raw_txn_parallel_upload; +#[path = "it_rdf_text_parity.rs"] +mod it_rdf_text_parity; #[path = "it_reserved_graph_writes.rs"] mod it_reserved_graph_writes; #[path = "it_stable_blank_nodes.rs"] @@ -57,3 +61,5 @@ mod it_txn_meta; mod it_update_wildcard_delete_indexed; #[path = "it_upsert_duplicate_ids_repro.rs"] mod it_upsert_duplicate_ids_repro; +#[path = "it_upsert_stored_retractions.rs"] +mod it_upsert_stored_retractions; diff --git a/fluree-db-api/tests/it_annotation_delete_named_graph.rs b/fluree-db-api/tests/it_annotation_delete_named_graph.rs new file mode 100644 index 0000000000..98473b4cd9 --- /dev/null +++ b/fluree-db-api/tests/it_annotation_delete_named_graph.rs @@ -0,0 +1,138 @@ +//! Deleting a named-graph annotation by id retracts its whole bundle and +//! its body in that graph. +//! +//! The by-id delete names the bundle's subject, predicate and object slots. +//! The cascade then cleaned up the annotation's body (LPG mode) through a +//! range read whose index-resident rows carry no graph, so the body's +//! retraction landed in the default graph and the body survived in its own +//! graph. And nothing retracted the bundle's `f:reifiesGraph` anchor, which +//! stayed behind describing nothing. Both now come from the stored facts of +//! the annotation, graph stamped. + +use crate::support; +use fluree_db_api::{FlureeBuilder, LedgerState}; +use fluree_db_core::comparator::IndexType; +use fluree_db_core::{range_with_overlay, FlakeValue, RangeMatch, RangeOptions, RangeTest}; +use serde_json::json; + +const EX: &str = "http://example.org/"; +const G1: &str = "http://example.org/g1"; + +/// Every current fact of `subject` in `graph` (None = default), as +/// `pred=value`, sorted — `f:reifies*` included. +async fn facts(ledger: &LedgerState, graph: Option<&str>, subject: &str) -> Vec { + let g_id = match graph { + None => 0, + Some(iri) => ledger + .snapshot + .graph_registry + .graph_id_for_iri(iri) + .expect("graph"), + }; + let s = ledger + .snapshot + .encode_iri(&format!("{EX}{subject}")) + .expect("subject"); + let flakes = range_with_overlay( + &ledger.snapshot, + g_id, + ledger.novelty.as_ref(), + IndexType::Spot, + RangeTest::Eq, + RangeMatch::new().with_subject(s), + RangeOptions::new().with_to_t(ledger.t()), + ) + .await + .expect("range"); + let mut out: Vec = flakes + .iter() + .filter(|f| f.op) + .map(|f| match &f.o { + FlakeValue::String(v) => format!("{}={v}", f.p.name), + FlakeValue::Ref(r) => format!("{}=<{}>", f.p.name, r.name), + other => format!("{}={other:?}", f.p.name), + }) + .collect(); + out.sort(); + out +} + +#[tokio::test] +async fn by_id_delete_of_a_named_graph_annotation_retracts_bundle_and_body_in_that_graph() { + for indexed in [false, true] { + let dir = tempfile::tempdir().expect("tempdir"); + let fluree = FlureeBuilder::file(dir.path().to_string_lossy().to_string()) + .build() + .expect("fluree"); + let id = format!("it/ann-delete-g1-{indexed}:main"); + let ledger = fluree.create_ledger(&id).await.unwrap(); + let mut ledger = fluree + .insert( + ledger, + &json!({ + "@context": {"ex": EX}, + "@graph": [{ + "@id": "ex:alice", "@graph": G1, + "ex:worksFor": { + "@id": "ex:acme", + "@annotation": {"@id": "ex:empA", "ex:role": "Engineer"} + } + }] + }), + ) + .await + .unwrap() + .ledger; + if indexed { + drop(ledger); + support::rebuild_and_publish_index(&fluree, &id).await; + ledger = fluree.ledger(&id).await.unwrap(); + } + let before = facts(&ledger, Some(G1), "empA").await; + assert!( + before.iter().any(|f| f.starts_with("reifiesGraph=")) + && before.iter().any(|f| f == "role=Engineer"), + "indexed={indexed}: precondition {before:?}" + ); + + let r = fluree + .update( + ledger, + &json!({ + "@context": {"ex": EX}, + "graph": G1, + "delete": { + "@id": "ex:alice", + "ex:worksFor": {"@id": "ex:acme", "@annotation": {"@id": "ex:empA"}} + }, + "opts": {"lpgEdgeLifecycle": true} + }), + ) + .await + .expect("by-id annotation delete"); + let mut ledger = r.ledger; + for phase in ["after the delete", "after the next index build"] { + assert_eq!( + facts(&ledger, Some(G1), "empA").await, + Vec::::new(), + "indexed={indexed}, {phase}: the bundle (anchor included) and the body are gone" + ); + assert_eq!( + facts(&ledger, Some(G1), "alice").await, + ["worksFor="], + "indexed={indexed}, {phase}: the base edge stays" + ); + if phase == "after the delete" { + drop(ledger); + support::rebuild_and_publish_index(&fluree, &id).await; + ledger = fluree.ledger(&id).await.unwrap(); + } + } + // Nothing of the annotation's was ever in the default graph, so + // nothing may be retracted there. + assert_eq!( + r.receipt.retract_count, 5, + "indexed={indexed}: three named slots, the anchor, and the body" + ); + } +} diff --git a/fluree-db-api/tests/it_delete_stored_facts.rs b/fluree-db-api/tests/it_delete_stored_facts.rs new file mode 100644 index 0000000000..d56a752097 --- /dev/null +++ b/fluree-db-api/tests/it_delete_stored_facts.rs @@ -0,0 +1,1503 @@ +//! A DELETE retracts only facts that are stored. +//! +//! A retraction of a fact that is not stored used to be committed anyway +//! (a "phantom"), with two effects: +//! - `DELETE DATA` of an absent triple committed a new `t` holding one +//! retraction, though `docs/transactions/retractions.md` says it is a +//! no-op; +//! - `DELETE {x} INSERT {x}` with `x` absent committed nothing and never +//! wrote `x`: the phantom retraction cancelled the insert, against SPARQL +//! 1.1 Update's `(DS − D) ∪ I`. +//! +//! Every DELETE intent that a WHERE triple does not witness is now matched +//! against the stored facts of its slot, and one that names none stages +//! nothing. Each case has SPARQL and JSON-LD twins (Cypher where its +//! lowering produces the shape), on a novelty-only ledger and an indexed +//! one. +//! +//! On an indexed ledger an intent also names a fact by its index key, which +//! keeps less than some terms carry: a big integer's XSD subtype, a +//! `dateTime`'s digits past the microsecond. + +use crate::support; +use fluree_db_api::policy_builder::build_policy_context_from_opts; +use fluree_db_api::{ + CommitRef, ConflictStrategy, Fluree, FlureeBuilder, GovernanceOptions, LedgerState, + PolicyContext, TransactResult, +}; +use fluree_db_core::comparator::IndexType; +use fluree_db_core::{range_with_overlay, FlakeValue, RangeMatch, RangeOptions, RangeTest}; +use serde_json::{json, Value as JsonValue}; + +const EX: &str = "http://example.org/ns/"; +const G1: &str = "http://example.org/g1"; + +fn ctx() -> JsonValue { + json!({"ex": EX, "xsd": "http://www.w3.org/2001/XMLSchema#"}) +} + +async fn sparql_update(fluree: &Fluree, ledger: LedgerState, body: &str) -> TransactResult { + let text = + format!("PREFIX ex: <{EX}>\nPREFIX xsd: \n{body}"); + let parsed = fluree_db_sparql::parse_sparql(&text); + assert!(!parsed.has_errors(), "{text}: {:?}", parsed.diagnostics); + let mut ns = fluree_db_transact::NamespaceRegistry::from_db(&ledger.snapshot); + let txn = fluree_db_transact::lower_sparql_update_ast( + &parsed.ast.expect("ast"), + &mut ns, + fluree_db_transact::TxnOpts::default(), + ) + .expect("lower"); + fluree + .stage_owned(ledger) + .txn(txn) + .execute() + .await + .expect("SPARQL update") +} + +async fn jsonld_update(fluree: &Fluree, ledger: LedgerState, mut txn: JsonValue) -> TransactResult { + txn["@context"] = ctx(); + fluree.update(ledger, &txn).await.expect("JSON-LD update") +} + +/// `SELECT ?o` over `?s ?p ?o` for `ex:s`, as `p=value[@lang]`, sorted. +async fn facts(fluree: &Fluree, ledger: &LedgerState) -> Vec { + let q = format!( + "SELECT ?p ?v ?l WHERE {{ <{EX}s> ?p ?o BIND(STR(?o) AS ?v) BIND(LANG(?o) AS ?l) }}" + ); + let result = support::query_sparql(fluree, ledger, &q) + .await + .expect("query"); + let json = result.to_jsonld(&ledger.snapshot).expect("jsonld"); + let cell = |c: &JsonValue| { + c.as_str() + .or_else(|| c.get("@id").and_then(JsonValue::as_str)) + .or_else(|| c.get("@value").and_then(JsonValue::as_str)) + .map(str::to_string) + .unwrap_or_else(|| c.to_string()) + }; + let mut rows: Vec = json + .as_array() + .expect("rows") + .iter() + .map(|r| { + let r = r.as_array().expect("row"); + let p = cell(&r[0]); + let p = p.rsplit(['/', '#', ':']).next().unwrap_or(&p).to_string(); + let lang = cell(&r[2]); + if lang.is_empty() || lang == "null" { + format!("{p}={}", cell(&r[1])) + } else { + format!("{p}={}@{lang}", cell(&r[1])) + } + }) + .collect(); + rows.sort(); + rows +} + +/// A file-backed ledger holding `seed`, optionally indexed. +async fn seeded( + name: &str, + seed: JsonValue, + indexed: bool, +) -> (tempfile::TempDir, Fluree, LedgerState) { + let dir = tempfile::tempdir().expect("tempdir"); + let fluree = FlureeBuilder::file(dir.path().to_string_lossy().to_string()) + .build() + .expect("file fluree"); + let id = format!("it/delete-stored-{name}-{indexed}:main"); + let ledger = fluree.create_ledger(&id).await.expect("create"); + let mut doc = seed; + doc["@context"] = ctx(); + let ledger = fluree.insert(ledger, &doc).await.expect("seed").ledger; + if !indexed { + return (dir, fluree, ledger); + } + drop(ledger); + support::rebuild_and_publish_index(&fluree, &id).await; + let ledger = fluree.ledger(&id).await.expect("reload"); + assert!(ledger.snapshot.range_provider.is_some()); + (dir, fluree, ledger) +} + +fn no_commit(r: &TransactResult, t_before: i64, what: &str) { + assert_eq!( + (r.receipt.flake_count, r.ledger.t()), + (0, t_before), + "{what}: expected no commit" + ); +} + +#[tokio::test] +async fn deleting_an_absent_fact_commits_nothing() { + for indexed in [false, true] { + let seed = json!({"@id": "ex:s", "ex:p": "present"}); + + let (_d, fluree, ledger) = seeded("absent-sparql", seed.clone(), indexed).await; + let t = ledger.t(); + let r = sparql_update(&fluree, ledger, "DELETE DATA { ex:s ex:p \"absent\" }").await; + no_commit(&r, t, &format!("SPARQL DELETE DATA (indexed={indexed})")); + assert_eq!(facts(&fluree, &r.ledger).await, ["p=present"]); + + let (_d, fluree, ledger) = seeded("absent-jsonld", seed.clone(), indexed).await; + let t = ledger.t(); + let r = jsonld_update( + &fluree, + ledger, + json!({"delete": {"@id": "ex:s", "ex:p": "absent"}}), + ) + .await; + no_commit(&r, t, &format!("JSON-LD delete (indexed={indexed})")); + + // A retraction of a present fact next to an absent one retracts + // only the present one. + let (_d, fluree, ledger) = seeded("absent-mixed", seed.clone(), indexed).await; + let r = sparql_update( + &fluree, + ledger, + "DELETE DATA { ex:s ex:p \"absent\" . ex:s ex:p \"present\" }", + ) + .await; + assert_eq!( + (r.receipt.retract_count, r.receipt.assert_count), + (1, 0), + "indexed={indexed}" + ); + assert!(facts(&fluree, &r.ledger).await.is_empty()); + + // Cypher `REMOVE n:Label` retracts `rdf:type Label`, a constant. + let (_d, fluree, ledger) = seeded("absent-cypher", seed.clone(), indexed).await; + let ledger = fluree + .transact_cypher(ledger, "CREATE (n:Person {name: 'n'})") + .await + .expect("cypher CREATE") + .ledger; + let t = ledger.t(); + let r = fluree + .transact_cypher(ledger, "MATCH (n:Person {name: 'n'}) REMOVE n:Missing") + .await + .expect("cypher REMOVE"); + no_commit( + &r, + t, + &format!("Cypher REMOVE of an absent label (indexed={indexed})"), + ); + } +} + +/// A delete that names a graph the ledger does not have commits nothing and +/// registers no graph, as in the default graph; it used to land an empty +/// commit that registered the graph. A graph an INSERT template names is still +/// registered when the WHERE matches nothing, delete beside it or not +/// (`no_match_update_naming_a_new_graph_commits_its_registration`). +#[tokio::test] +async fn a_delete_in_a_new_graph_commits_nothing() { + const NEW: &str = "http://example.org/new"; + let registered = |r: &TransactResult| r.ledger.snapshot.graph_registry.graph_id_for_iri(NEW); + for indexed in [false, true] { + let seed = json!({"@id": "ex:s", "ex:p": "present"}); + + let (_d, fluree, ledger) = seeded("new-graph-sparql", seed.clone(), indexed).await; + let t = ledger.t(); + let r = sparql_update( + &fluree, + ledger, + &format!("DELETE DATA {{ GRAPH <{NEW}> {{ ex:s ex:p \"present\" }} }}"), + ) + .await; + let what = format!("SPARQL DELETE DATA (indexed={indexed})"); + no_commit(&r, t, &what); + assert!(registered(&r).is_none(), "{what}: registered {NEW}"); + + let (_d, fluree, ledger) = seeded("new-graph-jsonld", seed.clone(), indexed).await; + let t = ledger.t(); + let r = jsonld_update( + &fluree, + ledger, + json!({"graph": NEW, "delete": {"@id": "ex:s", "ex:p": "present"}}), + ) + .await; + let what = format!("JSON-LD delete (indexed={indexed})"); + no_commit(&r, t, &what); + assert!(registered(&r).is_none(), "{what}: registered {NEW}"); + + let (_d, fluree, ledger) = seeded("new-graph-insert", seed.clone(), indexed).await; + let t = ledger.t(); + let r = sparql_update( + &fluree, + ledger, + &format!( + "DELETE {{ GRAPH <{NEW}> {{ ?s ex:q \"x\" }} }} \ + INSERT {{ GRAPH <{NEW}> {{ ?s ex:q \"y\" }} }} WHERE {{ ?s ex:p \"nothing\" }}" + ), + ) + .await; + let what = format!("DELETE and INSERT, no match (indexed={indexed})"); + assert_eq!( + (r.receipt.flake_count, r.ledger.t()), + (0, t + 1), + "{what}: the registration commits" + ); + assert!(registered(&r).is_some(), "{what}: {NEW} not registered"); + } +} + +/// A WHERE-driven DELETE removes every stored fact its rows name, list +/// entries holding the same value included: each identical row used to take +/// the first stored entry, so `DELETE WHERE` over `["v", "v", "w"]` left one +/// `"v"` behind, and over a plain `"v"` beside a list `["v"]` left the plain +/// one. A WHERE that repeats each row (a second pattern matching twice) +/// retracts each stored fact once. +#[tokio::test] +async fn a_where_driven_delete_removes_every_identical_list_entry() { + for indexed in [false, true] { + for (what, seed, delete, retracted) in [ + ( + "repeated list value", + json!({"@id": "ex:s", "ex:list": {"@list": ["v", "v", "w"]}}), + "DELETE WHERE { ex:s ex:list ?o }", + 3, + ), + ( + "plain value beside a list entry", + json!({"@id": "ex:s", "ex:list": ["v", {"@list": ["v"]}]}), + "DELETE WHERE { ex:s ex:list ?o }", + 2, + ), + ( + "rows repeated by another pattern", + json!({"@graph": [ + {"@id": "ex:s", "ex:list": {"@list": ["v", "v"]}}, + {"@id": "ex:x1", "ex:q": 1}, + {"@id": "ex:x2", "ex:q": 2} + ]}), + "DELETE { ex:s ex:list ?o } WHERE { ex:s ex:list ?o . ?x ex:q ?y }", + 2, + ), + ] { + let (_d, fluree, ledger) = seeded("identical-entries", seed, indexed).await; + let r = sparql_update(&fluree, ledger, delete).await; + let what = format!("{what} (indexed={indexed})"); + assert_eq!(r.receipt.retract_count, retracted, "{what}"); + assert!(facts(&fluree, &r.ledger).await.is_empty(), "{what}"); + } + } +} + +#[tokio::test] +async fn delete_insert_of_an_absent_fact_inserts_it() { + for indexed in [false, true] { + let seed = json!({"@id": "ex:s", "ex:other": "o"}); + + let (_d, fluree, ledger) = seeded("di-sparql", seed.clone(), indexed).await; + let r = sparql_update( + &fluree, + ledger, + "DELETE { ex:s ex:p \"x\" } INSERT { ex:s ex:p \"x\" } WHERE {}", + ) + .await; + assert_eq!( + facts(&fluree, &r.ledger).await, + ["other=o", "p=x"], + "SPARQL, x absent (indexed={indexed})" + ); + // x present now: the same update is a no-op. + let t = r.ledger.t(); + let r = sparql_update( + &fluree, + r.ledger, + "DELETE { ex:s ex:p \"x\" } INSERT { ex:s ex:p \"x\" } WHERE {}", + ) + .await; + no_commit(&r, t, &format!("SPARQL, x present (indexed={indexed})")); + assert_eq!(facts(&fluree, &r.ledger).await, ["other=o", "p=x"]); + + let (_d, fluree, ledger) = seeded("di-jsonld", seed.clone(), indexed).await; + let txn = json!({ + "delete": {"@id": "ex:s", "ex:p": "x"}, + "insert": {"@id": "ex:s", "ex:p": "x"} + }); + let r = jsonld_update(&fluree, ledger, txn.clone()).await; + assert_eq!( + facts(&fluree, &r.ledger).await, + ["other=o", "p=x"], + "JSON-LD, x absent (indexed={indexed})" + ); + let t = r.ledger.t(); + let r = jsonld_update(&fluree, r.ledger, txn).await; + no_commit(&r, t, &format!("JSON-LD, x present (indexed={indexed})")); + } +} + +#[tokio::test] +async fn delete_data_names_terms_exactly() { + for indexed in [false, true] { + // A language tag compares case-insensitively; the stored tag is the + // one retracted. + let (_d, fluree, ledger) = seeded( + "lang", + json!({"@id": "ex:s", "ex:label": {"@value": "a", "@language": "en"}}), + indexed, + ) + .await; + let r = sparql_update(&fluree, ledger, "DELETE DATA { ex:s ex:label \"a\"@EN }").await; + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "indexed={indexed}" + ); + + // The datatype is part of the term: an xsd:int is not deleted by an + // xsd:integer of the same value. + let (_d, fluree, ledger) = seeded( + "dt", + json!({"@id": "ex:s", "ex:n": {"@value": "1", "@type": "xsd:int"}}), + indexed, + ) + .await; + let t = ledger.t(); + let r = sparql_update(&fluree, ledger, "DELETE DATA { ex:s ex:n 1 }").await; + no_commit( + &r, + t, + &format!("xsd:integer vs stored xsd:int (indexed={indexed})"), + ); + assert_eq!(facts(&fluree, &r.ledger).await, ["n=1"]); + // The same term deletes it (spelled in JSON-LD here; + // `delete_data_of_typed_literals_deletes_them` covers SPARQL). + let r = jsonld_update( + &fluree, + r.ledger, + json!({"delete": {"@id": "ex:s", "ex:n": {"@value": "1", "@type": "xsd:int"}}}), + ) + .await; + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "indexed={indexed}" + ); + } +} + +/// SPARQL `DELETE DATA` of typed literals (`xsd:int`, `xsd:long`, +/// `xsd:dateTime`). SPARQL UPDATE used to keep their lexical form as a +/// string, so the intent named no stored term and the delete committed +/// phantom retractions that removed nothing. The lowering now coerces them +/// as JSON-LD does (#1988), the intent names the stored term exactly, and +/// the resolver deletes it. +#[tokio::test] +async fn delete_data_of_typed_literals_deletes_them() { + for indexed in [false, true] { + let (_d, fluree, ledger) = seeded( + "typed", + json!({ + "@id": "ex:s", + "ex:n": {"@value": "1", "@type": "xsd:int"}, + "ex:l": {"@value": "5", "@type": "xsd:long"}, + "ex:d": {"@value": "2020-01-01T00:00:00Z", "@type": "xsd:dateTime"} + }), + indexed, + ) + .await; + let r = sparql_update( + &fluree, + ledger, + "DELETE DATA { ex:s ex:n \"1\"^^xsd:int . ex:s ex:l \"5\"^^xsd:long . \ + ex:s ex:d \"2020-01-01T00:00:00Z\"^^xsd:dateTime }", + ) + .await; + assert_eq!(r.receipt.retract_count, 3, "indexed={indexed}"); + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "indexed={indexed}" + ); + } +} + +/// The SPARQL and JSON-LD twins of the upsert-over-languages case: a +/// same-shape DELETE/INSERT WHERE replaces every language, because each +/// WHERE row is the decode of the stored fact, tag included. +#[tokio::test] +async fn delete_insert_where_replaces_every_language() { + for indexed in [false, true] { + let seed = json!({ + "@id": "ex:s", + "ex:label": [{"@value": "a", "@language": "en"}, {"@value": "a", "@language": "fr"}] + }); + let (_d, fluree, ledger) = seeded("langs-sparql", seed.clone(), indexed).await; + let r = sparql_update( + &fluree, + ledger, + "DELETE { ex:s ex:label ?o } INSERT { ex:s ex:label \"b\"@en } WHERE { ex:s ex:label ?o }", + ) + .await; + assert_eq!( + facts(&fluree, &r.ledger).await, + ["label=b@en"], + "indexed={indexed}" + ); + + let (_d, fluree, ledger) = seeded("langs-jsonld", seed, indexed).await; + let r = jsonld_update( + &fluree, + ledger, + json!({ + "where": {"@id": "ex:s", "ex:label": "?o"}, + "delete": {"@id": "ex:s", "ex:label": "?o"}, + "insert": {"@id": "ex:s", "ex:label": {"@value": "b", "@language": "en"}} + }), + ) + .await; + assert_eq!( + facts(&fluree, &r.ledger).await, + ["label=b@en"], + "indexed={indexed}" + ); + } +} + +/// A big integer typed with an XSD subtype decodes from the index with a +/// lossy datatype (`NUM_BIG` stores none). A witnessed DELETE WHERE row +/// carries that decode and still deletes the value through the shared +/// storage key, as before. On a list-bearing ledger the row also goes +/// through the resolver for list positions; this pins that the exact +/// datatype rule does not drop it there. +async fn numbig_rows(fluree: &Fluree, ledger: &LedgerState) -> usize { + let q = format!("PREFIX ex: <{EX}>\nSELECT ?o WHERE {{ GRAPH <{G1}> {{ ?s ex:n ?o }} }}"); + let r = support::query_sparql(fluree, ledger, &q).await.expect("q"); + r.to_jsonld(&ledger.snapshot) + .expect("jsonld") + .as_array() + .map_or(0, Vec::len) +} + +#[tokio::test] +async fn witnessed_delete_of_an_indexed_big_integer_in_a_named_graph() { + for with_list in [false, true] { + let mut graph = vec![json!({ + "@id": "ex:s", "@graph": G1, + "ex:n": {"@value": "18446744073709551615", "@type": "xsd:unsignedLong"} + })]; + if with_list { + graph.push(json!({"@id": "ex:other", "ex:items": {"@list": ["a", "b"]}})); + } + for surface in ["sparql", "jsonld"] { + let (_d, fluree, ledger) = seeded( + &format!("numbig-{surface}-{with_list}"), + json!({"@graph": graph.clone()}), + true, + ) + .await; + assert_eq!(ledger.snapshot.has_list_meta, Some(with_list)); + assert_eq!(numbig_rows(&fluree, &ledger).await, 1, "precondition"); + let r = if surface == "sparql" { + sparql_update( + &fluree, + ledger, + &format!("DELETE WHERE {{ GRAPH <{G1}> {{ ?s ex:n ?o }} }}"), + ) + .await + } else { + jsonld_update( + &fluree, + ledger, + json!({ + "where": [["graph", G1, {"@id": "?s", "ex:n": "?o"}]], + "delete": [["graph", G1, {"@id": "?s", "ex:n": "?o"}]] + }), + ) + .await + }; + assert_eq!( + numbig_rows(&fluree, &r.ledger).await, + 0, + "{surface}, with_list={with_list}" + ); + } + } +} + +// ============================================================================= +// Indexed facts are named by their index key +// ============================================================================= + +/// A big integer past the range of `i64`. The persisted index keys such a +/// value by the value alone: its XSD subtype is not stored. +const BIG: &str = "123456789012345678901234567890"; + +/// The subtypes the big-integer cells cover, each with a big value it can +/// hold. +fn big_subtypes() -> [(&'static str, String); 4] { + [ + ("nonNegativeInteger", BIG.to_string()), + ("positiveInteger", BIG.to_string()), + ("negativeInteger", format!("-{BIG}")), + ("unsignedLong", "18446744073709551615".to_string()), + ] +} + +fn node_seed() -> JsonValue { + json!({"@id": "ex:s", "@type": "ex:Node"}) +} + +const NODE_ONLY: [&str; 1] = ["type=http://example.org/ns/Node"]; + +/// A DELETE that names a big integer under its declared subtype deletes it, +/// on a novelty-only ledger and on an indexed one, where the index decodes +/// the value as `xsd:integer`: `DELETE DATA`, a JSON-LD delete, and a +/// constant `DELETE` template alike. +#[tokio::test] +async fn constant_deletes_name_big_integers_under_their_subtype() { + let mut seed = node_seed(); + let mut del = json!({"@id": "ex:s"}); + let mut triples = Vec::new(); + for (dt, v) in big_subtypes() { + let term = json!({"@value": v, "@type": format!("xsd:{dt}")}); + seed[format!("ex:{dt}")] = term.clone(); + del[format!("ex:{dt}")] = term; + triples.push(format!("ex:s ex:{dt} \"{v}\"^^xsd:{dt}")); + } + let triples = triples.join(" . "); + for indexed in [false, true] { + for form in ["data", "jsonld", "template"] { + let (_d, fluree, ledger) = seeded(&format!("big-{form}"), seed.clone(), indexed).await; + let r = match form { + "data" => { + sparql_update(&fluree, ledger, &format!("DELETE DATA {{ {triples} }}")).await + } + "template" => { + sparql_update( + &fluree, + ledger, + &format!("DELETE {{ {triples} }} WHERE {{ ex:s a ex:Node }}"), + ) + .await + } + _ => jsonld_update(&fluree, ledger, json!({"delete": del.clone()})).await, + }; + assert_eq!( + facts(&fluree, &r.ledger).await, + NODE_ONLY, + "{form}, indexed={indexed}" + ); + assert_eq!(r.receipt.retract_count, 4, "{form}, indexed={indexed}"); + } + } +} + +/// A novelty retraction of an indexed fact under another identity with the +/// same index key leaves no ghost: a revert retracts a big integer under its +/// declared subtype, which the index row does not keep, and queries stop +/// seeing the value. The stored-fact read used to keep reporting it, so a +/// later upsert of the same value cancelled against it and committed +/// nothing, and the value stayed gone until the next reindex. +#[tokio::test] +async fn a_reverted_indexed_big_integer_can_be_written_again() { + for dt in ["integer", "nonNegativeInteger"] { + let dir = tempfile::tempdir().expect("tempdir"); + let fluree = FlureeBuilder::file(dir.path().to_string_lossy().to_string()) + .build() + .expect("file fluree"); + let name = format!("it/delete-stored-revert-{dt}"); + let id = format!("{name}:main"); + let ledger = fluree.create_ledger(&id).await.expect("create"); + let ledger = fluree + .insert( + ledger, + &json!({"@context": ctx(), "@id": "ex:s", "ex:other": 1}), + ) + .await + .expect("seed") + .ledger; + let value = json!({"@context": ctx(), "@id": "ex:s", + "ex:n": {"@value": BIG, "@type": format!("xsd:{dt}")}}); + let inserted = fluree.insert(ledger, &value).await.expect("insert"); + let commit = inserted.receipt.commit_id.clone(); + drop(inserted); + support::rebuild_and_publish_index(&fluree, &id).await; + fluree + .revert_commit( + &name, + "main", + CommitRef::Exact(commit), + ConflictStrategy::TakeSource, + ) + .await + .expect("revert"); + let ledger = fluree.ledger(&id).await.expect("reload"); + assert_eq!( + facts(&fluree, &ledger).await, + ["other=1"], + "xsd:{dt}: reverted" + ); + + let r = fluree.upsert(ledger, &value).await.expect("upsert"); + assert_eq!( + (r.receipt.assert_count, r.receipt.retract_count), + (1, 0), + "xsd:{dt}: the upsert writes the value" + ); + assert_eq!( + facts(&fluree, &r.ledger).await, + [format!("n={BIG}"), "other=1".to_string()], + "xsd:{dt}: written again" + ); + } +} + +/// A value bound by OPTIONAL, UNION or BIND is not a witness, so its DELETE +/// row is matched against the stored facts of its slot. The row carries the +/// query's decode of the value, which on an indexed ledger has lost a +/// declared subtype, and the row still names the fact. +/// +/// Not covered: BIND on a novelty-only ledger with a subtype. BIND gives a +/// big integer `xsd:integer` whatever its declared subtype, so there the row +/// names a term that is not stored. That is a query-side gap, on `main` too. +#[tokio::test] +async fn where_bound_rows_name_big_integers() { + for indexed in [false, true] { + for dt in ["integer", "nonNegativeInteger"] { + let mut seed = node_seed(); + seed["ex:n"] = json!({"@value": BIG, "@type": format!("xsd:{dt}")}); + for (shape, body) in [ + ( + "optional", + "DELETE { ex:s ex:n ?o } WHERE { ex:s a ex:Node OPTIONAL { ex:s ex:n ?o } }", + ), + ( + "union", + "DELETE { ex:s ex:n ?o } WHERE { { ex:s ex:n ?o } UNION { ex:s ex:none ?o } }", + ), + ( + "bind", + "DELETE { ex:s ex:n ?o } WHERE { ex:s ex:n ?v BIND(?v AS ?o) }", + ), + ] { + if shape == "bind" && dt != "integer" && !indexed { + continue; + } + let (_d, fluree, ledger) = + seeded(&format!("big-{shape}-{dt}"), seed.clone(), indexed).await; + let r = sparql_update(&fluree, ledger, body).await; + assert_eq!( + facts(&fluree, &r.ledger).await, + NODE_ONLY, + "{shape}, xsd:{dt}, indexed={indexed}" + ); + } + let (_d, fluree, ledger) = + seeded(&format!("big-optional-jsonld-{dt}"), seed.clone(), indexed).await; + let r = jsonld_update( + &fluree, + ledger, + json!({ + "where": [{"@id": "ex:s", "@type": "ex:Node"}, ["optional", {"@id": "ex:s", "ex:n": "?o"}]], + "delete": {"@id": "ex:s", "ex:n": "?o"} + }), + ) + .await; + assert_eq!( + facts(&fluree, &r.ledger).await, + NODE_ONLY, + "JSON-LD optional, xsd:{dt}, indexed={indexed}" + ); + } + } +} + +/// Cypher `SET` replaces a big-integer property and `DETACH DELETE` removes +/// it with its node, on both lanes: both read the old value through an +/// OPTIONAL row. +#[tokio::test] +async fn cypher_set_and_detach_delete_reach_big_integers() { + for indexed in [false, true] { + let mut seed = node_seed(); + seed["ex:big"] = json!({"@value": BIG, "@type": "xsd:integer"}); + + let (_d, fluree, ledger) = seeded("big-cypher-set", seed.clone(), indexed).await; + let r = fluree + .transact_cypher( + ledger, + "MATCH (n:`http://example.org/ns/Node`) SET n.`http://example.org/ns/big` = 5", + ) + .await + .expect("cypher SET"); + assert_eq!( + facts(&fluree, &r.ledger).await, + ["big=5", "type=http://example.org/ns/Node"], + "SET, indexed={indexed}" + ); + + let (_d, fluree, ledger) = seeded("big-cypher-detach", seed, indexed).await; + let r = fluree + .transact_cypher( + ledger, + "MATCH (n:`http://example.org/ns/Node`) DETACH DELETE n", + ) + .await + .expect("cypher DETACH DELETE"); + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "DETACH DELETE, indexed={indexed}: {:?}", + facts(&fluree, &r.ledger).await + ); + } +} + +/// A `dateTime` or `time` written past the microsecond is deleted by the +/// same literal on both lanes; the index keeps microseconds. +#[tokio::test] +async fn delete_data_names_sub_microsecond_temporals() { + let seed = json!({ + "@id": "ex:s", + "ex:d": {"@value": "2020-01-01T00:00:00.123456789Z", "@type": "xsd:dateTime"}, + "ex:t": {"@value": "12:00:00.123456789", "@type": "xsd:time"} + }); + for indexed in [false, true] { + let (_d, fluree, ledger) = seeded("sub-us-sparql", seed.clone(), indexed).await; + let r = sparql_update( + &fluree, + ledger, + "DELETE DATA { ex:s ex:d \"2020-01-01T00:00:00.123456789Z\"^^xsd:dateTime . \ + ex:s ex:t \"12:00:00.123456789\"^^xsd:time }", + ) + .await; + assert_eq!(r.receipt.retract_count, 2, "SPARQL, indexed={indexed}"); + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "SPARQL, indexed={indexed}" + ); + + let (_d, fluree, ledger) = seeded("sub-us-jsonld", seed.clone(), indexed).await; + let mut del = seed.clone(); + del["@id"] = json!("ex:s"); + let r = jsonld_update(&fluree, ledger, json!({"delete": del})).await; + assert_eq!(r.receipt.retract_count, 2, "JSON-LD, indexed={indexed}"); + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "JSON-LD, indexed={indexed}" + ); + } +} + +/// The index key names one value: a DELETE of another big integer under +/// the same subtype, or of the same digits as a decimal, names nothing and +/// commits nothing. +#[tokio::test] +async fn a_different_big_value_is_not_deleted() { + let mut seed = node_seed(); + seed["ex:n"] = json!({"@value": BIG, "@type": "xsd:nonNegativeInteger"}); + for indexed in [false, true] { + let (_d, fluree, ledger) = seeded("big-other", seed.clone(), indexed).await; + let t = ledger.t(); + let r = sparql_update( + &fluree, + ledger, + &format!( + "DELETE DATA {{ ex:s ex:n \"{BIG}1\"^^xsd:nonNegativeInteger . \ + ex:s ex:n \"{BIG}.0\"^^xsd:decimal }}" + ), + ) + .await; + no_commit(&r, t, &format!("indexed={indexed}")); + assert_eq!( + facts(&fluree, &r.ledger).await, + [ + format!("n={BIG}"), + "type=http://example.org/ns/Node".to_string() + ], + "indexed={indexed}" + ); + } +} + +/// The index keys a big integer per graph and predicate, so a named graph's +/// fact is named by its own graph's key. +#[tokio::test] +async fn a_delete_names_a_big_integer_in_a_named_graph() { + let seed = json!({"@graph": [{ + "@id": "ex:s", "@graph": G1, + "ex:n": {"@value": BIG, "@type": "xsd:nonNegativeInteger"} + }]}); + for indexed in [false, true] { + let (_d, fluree, ledger) = seeded("big-graph", seed.clone(), indexed).await; + assert_eq!(numbig_rows(&fluree, &ledger).await, 1, "precondition"); + let r = sparql_update( + &fluree, + ledger, + &format!( + "DELETE DATA {{ GRAPH <{G1}> {{ ex:s ex:n \"{BIG}\"^^xsd:nonNegativeInteger }} }}" + ), + ) + .await; + assert_eq!(r.receipt.retract_count, 1, "indexed={indexed}"); + assert_eq!( + numbig_rows(&fluree, &r.ledger).await, + 0, + "indexed={indexed}" + ); + } +} + +// ============================================================================= +// Modify policy judges every delete a transaction asks for +// ============================================================================= + +async fn policy(ledger: &LedgerState, opts: GovernanceOptions) -> PolicyContext { + build_policy_context_from_opts( + &ledger.snapshot, + ledger.novelty.as_ref(), + Some(ledger.novelty.as_ref()), + ledger.t(), + &opts, + &[0], + ) + .await + .expect("build policy context") +} + +/// The identity may view and modify `ex:name` and nothing else. +async fn name_only(ledger: &LedgerState) -> PolicyContext { + policy( + ledger, + GovernanceOptions { + policy: Some(json!([{ + "@id": "http://example.org/ns/nameOnly", + "f:onProperty": [{"@id": "http://example.org/ns/name"}], + "f:action": [{"@id": "f:view"}, {"@id": "f:modify"}], + "f:allow": true + }])), + default_allow: Some(false), + ..Default::default() + }, + ) + .await +} + +/// The identity may modify everything but `ex:ssn`. +async fn all_but_ssn(ledger: &LedgerState) -> PolicyContext { + policy( + ledger, + GovernanceOptions { + policy: Some(json!([{ + "@id": "http://example.org/ns/denySsn", + "f:required": true, + "f:onProperty": [{"@id": "http://example.org/ns/ssn"}], + "f:action": "f:modify", + "f:allow": false + }])), + default_allow: Some(true), + ..Default::default() + }, + ) + .await +} + +async fn sparql_update_as( + fluree: &Fluree, + ledger: LedgerState, + body: &str, + policy: PolicyContext, +) -> Result { + let text = + format!("PREFIX ex: <{EX}>\nPREFIX xsd: \n{body}"); + let parsed = fluree_db_sparql::parse_sparql(&text); + assert!(!parsed.has_errors(), "{text}: {:?}", parsed.diagnostics); + let mut ns = fluree_db_transact::NamespaceRegistry::from_db(&ledger.snapshot); + let txn = fluree_db_transact::lower_sparql_update_ast( + &parsed.ast.expect("ast"), + &mut ns, + fluree_db_transact::TxnOpts::default(), + ) + .expect("lower"); + fluree + .stage_owned(ledger) + .txn(txn) + .policy(policy) + .execute() + .await +} + +fn salary_seed() -> JsonValue { + json!({"@id": "ex:alice", "ex:name": "Alice", "ex:salary": 100}) +} + +/// A delete the identity may not perform is refused with one answer whether +/// or not the value it names is stored: here `ex:salary 100` is stored and +/// 999 and 50 are not. +#[tokio::test] +async fn delete_refused_by_policy_regardless_of_stored_value() { + for indexed in [false, true] { + for body in [ + "DELETE DATA { ex:alice ex:salary 100 }", + "DELETE DATA { ex:alice ex:salary 999 }", + "DELETE DATA { ex:alice ex:salary 50 }", + "DELETE { ?s ex:salary 100 } WHERE { ?s ex:name ?n }", + "DELETE { ?s ex:salary 999 } WHERE { ?s ex:name ?n }", + ] { + let (_d, fluree, ledger) = seeded("policy-sparql", salary_seed(), indexed).await; + let policy = name_only(&ledger).await; + let err = sparql_update_as(&fluree, ledger, body, policy) + .await + .map(|r| r.receipt.flake_count) + .expect_err(body); + assert!( + err.to_string() + .contains("Policy enforcement prevents modification"), + "{body}, indexed={indexed}: {err}" + ); + } + for salary in [100, 999] { + let (_d, fluree, ledger) = seeded("policy-jsonld", salary_seed(), indexed).await; + let policy = name_only(&ledger).await; + let txn = json!({ + "@context": ctx(), + "delete": {"@id": "ex:alice", "ex:salary": salary} + }); + let err = fluree + .stage_owned(ledger) + .update(&txn) + .policy(policy) + .execute() + .await + .map(|r| r.receipt.flake_count) + .expect_err("JSON-LD delete"); + assert!( + err.to_string() + .contains("Policy enforcement prevents modification"), + "JSON-LD {salary}, indexed={indexed}: {err}" + ); + } + } +} + +/// An identity the policy lets modify the value: deleting it retracts it, +/// and deleting a value that is not stored commits nothing. +#[tokio::test] +async fn a_permitted_delete_of_an_absent_value_commits_nothing() { + for indexed in [false, true] { + let (_d, fluree, ledger) = seeded("policy-allowed", salary_seed(), indexed).await; + let t = ledger.t(); + let policy = all_but_ssn(&ledger).await; + let r = sparql_update_as( + &fluree, + ledger, + "DELETE DATA { ex:alice ex:salary 999 }", + policy, + ) + .await + .expect("permitted"); + no_commit(&r, t, &format!("absent value, indexed={indexed}")); + + let policy = all_but_ssn(&r.ledger).await; + let r = sparql_update_as( + &fluree, + r.ledger, + "DELETE DATA { ex:alice ex:salary 100 }", + policy, + ) + .await + .expect("permitted"); + assert_eq!(r.receipt.retract_count, 1, "indexed={indexed}"); + } +} + +// ============================================================================= +// Modify policy judges a restated value the identity cannot view +// ============================================================================= + +/// `ex:alice` with a plain value, a language-tagged one and a list. +fn restate_seed() -> JsonValue { + json!({ + "@id": "ex:alice", + "ex:name": "Alice", + "ex:salary": 100, + "ex:label": {"@value": "Al", "@language": "en"}, + "ex:scores": {"@list": [1, 2, 3]} + }) +} + +/// The identity may view everything and modify only `ex:name`. +async fn view_all_modify_name(ledger: &LedgerState) -> PolicyContext { + policy( + ledger, + GovernanceOptions { + policy: Some(json!([ + { + "@id": "http://example.org/ns/viewAll", + "@type": "f:AccessPolicy", + "f:action": "f:view", + "f:allow": true + }, + { + "@id": "http://example.org/ns/modifyName", + "@type": "f:AccessPolicy", + "f:onProperty": [{"@id": "http://example.org/ns/name"}], + "f:action": "f:modify", + "f:allow": true + } + ])), + default_allow: Some(false), + ..Default::default() + }, + ) + .await +} + +/// Upserts that restate a stored value, and their twins that write a value +/// that is not stored. +fn restating_upserts() -> Vec<(&'static str, JsonValue)> { + vec![ + ( + "plain, stored", + json!({"@id": "ex:alice", "ex:salary": 100}), + ), + ( + "plain, not stored", + json!({"@id": "ex:alice", "ex:salary": 999}), + ), + ( + "language-tagged, stored", + json!({"@id": "ex:alice", "ex:label": {"@value": "Al", "@language": "en"}}), + ), + ( + "language-tagged, not stored", + json!({"@id": "ex:alice", "ex:label": {"@value": "Bo", "@language": "en"}}), + ), + ( + "list, stored", + json!({"@id": "ex:alice", "ex:scores": {"@list": [1, 2, 3]}}), + ), + ( + "list, not stored", + json!({"@id": "ex:alice", "ex:scores": {"@list": [1, 2, 4]}}), + ), + ] +} + +async fn upsert_as( + fluree: &Fluree, + ledger: LedgerState, + mut doc: JsonValue, + policy: Option, +) -> Result { + doc["@context"] = ctx(); + let builder = fluree.stage_owned(ledger).upsert(&doc); + match policy { + Some(policy) => builder.policy(policy).execute().await, + None => builder.execute().await, + } +} + +/// `None` when `result` is the modify-policy refusal, else what happened. +fn not_refused( + result: Result, + what: &str, +) -> Option { + match result { + Ok(r) => Some(format!( + "{what}: committed {} flakes", + r.receipt.flake_count + )), + Err(e) + if e.to_string() + .contains("Policy enforcement prevents modification") => + { + None + } + Err(e) => Some(format!("{what}: {e}")), + } +} + +/// An upsert the identity may not perform is refused with one answer +/// whether it restates a stored value or writes one that is not stored: +/// plain, language-tagged and list values, on JSON-LD and Turtle. The +/// identity may neither view nor modify them. +#[tokio::test] +async fn upsert_refused_by_policy_regardless_of_stored_value() { + let mut failures = Vec::new(); + for indexed in [false, true] { + for (what, doc) in restating_upserts() { + let (_d, fluree, ledger) = seeded("restate-upsert", restate_seed(), indexed).await; + let policy = name_only(&ledger).await; + let r = upsert_as(&fluree, ledger, doc, Some(policy)).await; + failures.extend(not_refused( + r, + &format!("JSON-LD {what}, indexed={indexed}"), + )); + } + for salary in [100, 999] { + let (_d, fluree, ledger) = seeded("restate-turtle", restate_seed(), indexed).await; + let policy = name_only(&ledger).await; + let ttl = format!("@prefix ex: <{EX}> .\nex:alice ex:salary {salary} .\n"); + let r = fluree + .stage_owned(ledger) + .upsert_turtle(&ttl) + .policy(policy) + .execute() + .await; + failures.extend(not_refused( + r, + &format!("Turtle {salary}, indexed={indexed}"), + )); + } + } + assert!(failures.is_empty(), "{failures:#?}"); +} + +/// A DELETE and INSERT of one value is refused with one answer whether or +/// not the value is stored, on SPARQL and JSON-LD; so is a graph sync whose +/// payload restates a graph the identity cannot view. +#[tokio::test] +async fn delete_insert_of_one_value_refused_by_policy_regardless_of_stored_value() { + let mut failures = Vec::new(); + for indexed in [false, true] { + for salary in [100, 999] { + let (_d, fluree, ledger) = seeded("restate-sparql", restate_seed(), indexed).await; + let policy = name_only(&ledger).await; + let body = format!( + "DELETE {{ ?s ex:salary {salary} }} INSERT {{ ?s ex:salary {salary} }} \ + WHERE {{ ?s ex:name \"Alice\" }}" + ); + let r = sparql_update_as(&fluree, ledger, &body, policy).await; + failures.extend(not_refused( + r, + &format!("SPARQL {salary}, indexed={indexed}"), + )); + + let (_d, fluree, ledger) = seeded("restate-jsonld", restate_seed(), indexed).await; + let policy = name_only(&ledger).await; + let txn = json!({ + "@context": ctx(), + "where": {"@id": "?s", "ex:name": "Alice"}, + "delete": {"@id": "?s", "ex:salary": salary}, + "insert": {"@id": "?s", "ex:salary": salary} + }); + let r = fluree + .stage_owned(ledger) + .update(&txn) + .policy(policy) + .execute() + .await; + failures.extend(not_refused( + r, + &format!("JSON-LD {salary}, indexed={indexed}"), + )); + } + } + for salary in [100, 999] { + let fluree = FlureeBuilder::memory().build_memory(); + let ledger = fluree + .create_ledger("it/delete-stored-restate-sync:main") + .await + .expect("create"); + let payload = |salary: i64| { + json!({ + "@context": ctx(), + "@graph": [{"@id": "ex:alice", "ex:name": "Alice", "ex:salary": salary}] + }) + }; + let seed = payload(100); + let ledger = fluree + .stage_owned(ledger) + .sync_graph(G1, &seed) + .execute() + .await + .expect("seed the graph") + .ledger; + let policy = name_only(&ledger).await; + let next = payload(salary); + let r = fluree + .stage_owned(ledger) + .sync_graph(G1, &next) + .policy(policy) + .execute() + .await; + failures.extend(not_refused(r, &format!("graph sync {salary}"))); + } + assert!(failures.is_empty(), "{failures:#?}"); +} + +/// Restating a value the identity can view commits nothing, without modify +/// permission, as it did before; a changed value is still refused. As root, +/// a restatement commits nothing either. +/// A transfer judges a fact the destination already holds as a restatement: +/// `ADD` or `COPY` of a value the identity can neither view nor modify is +/// refused with one answer whether or not the destination holds it. A held +/// fact used to stage nothing and commit, so the outcome told the identity +/// what the destination held. +#[tokio::test] +async fn transfer_refused_by_policy_regardless_of_destination_contents() { + const G2: &str = "http://example.org/g2"; + for indexed in [false, true] { + for dest_holds in [true, false] { + for verb in ["ADD", "COPY"] { + let dir = tempfile::tempdir().expect("tempdir"); + let fluree = FlureeBuilder::file(dir.path().to_string_lossy().to_string()) + .build() + .expect("file fluree"); + let id = format!("it/delete-stored-transfer-{verb}-{dest_holds}-{indexed}:main"); + let ledger = fluree.create_ledger(&id).await.expect("create"); + let dest = if dest_holds { + "ex:alice ex:salary 100" + } else { + "ex:alice ex:name \"Alice\"" + }; + let ledger = sparql_update( + &fluree, + ledger, + &format!( + "INSERT DATA {{ GRAPH <{G1}> {{ ex:alice ex:salary 100 }} \ + GRAPH <{G2}> {{ {dest} }} }}" + ), + ) + .await + .ledger; + let ledger = if indexed { + drop(ledger); + support::rebuild_and_publish_index(&fluree, &id).await; + fluree.ledger(&id).await.expect("reload") + } else { + ledger + }; + let policy = name_only(&ledger).await; + let body = format!("{verb} GRAPH <{G1}> TO GRAPH <{G2}>"); + let err = sparql_update_as(&fluree, ledger, &body, policy) + .await + .map(|r| r.receipt.flake_count) + .expect_err(&body); + assert!( + err.to_string() + .contains("Policy enforcement prevents modification"), + "{body}, destination holds it={dest_holds}, indexed={indexed}: {err}" + ); + } + } + } +} + +#[tokio::test] +async fn restating_a_viewable_value_commits_nothing_without_modify() { + for indexed in [false, true] { + for (what, doc) in restating_upserts() { + let stored = what.ends_with(", stored"); + let (_d, fluree, ledger) = seeded("restate-viewable", restate_seed(), indexed).await; + let t = ledger.t(); + let policy = view_all_modify_name(&ledger).await; + let r = upsert_as(&fluree, ledger, doc.clone(), Some(policy)).await; + let what = format!("{what}, indexed={indexed}"); + if stored { + no_commit(&r.expect(&what), t, &what); + } else { + assert_eq!(not_refused(r, &what), None); + } + if stored { + let (_d, fluree, ledger) = seeded("restate-root", restate_seed(), indexed).await; + let t = ledger.t(); + let r = upsert_as(&fluree, ledger, doc, None).await; + no_commit(&r.expect(&what), t, &format!("root, {what}")); + } + } + let (_d, fluree, ledger) = seeded("restate-viewable-sparql", restate_seed(), indexed).await; + let t = ledger.t(); + let policy = view_all_modify_name(&ledger).await; + let r = sparql_update_as( + &fluree, + ledger, + "DELETE { ?s ex:salary 100 } INSERT { ?s ex:salary 100 } WHERE { ?s ex:name \"Alice\" }", + policy, + ) + .await + .expect("restating a viewable value"); + no_commit(&r, t, &format!("SPARQL, indexed={indexed}")); + } +} + +// ============================================================================= +// A witness reads the graph its template writes +// ============================================================================= + +/// The strings stored as `ex:s ex:p` in the graph the ledger has under +/// `iri` (`None`: the default graph), read at the flake level: a query's +/// `GRAPH ` may resolve the name another way. +async fn graph_objects(ledger: &LedgerState, iri: Option<&str>) -> Vec { + let g_id = match iri { + None => 0, + Some(iri) => ledger + .snapshot + .graph_registry + .graph_id_for_iri(iri) + .expect("graph"), + }; + let s = ledger.snapshot.encode_iri(&format!("{EX}s")).expect("s"); + let p = ledger.snapshot.encode_iri(&format!("{EX}p")).expect("p"); + let flakes = range_with_overlay( + &ledger.snapshot, + g_id, + ledger.novelty.as_ref(), + IndexType::Spot, + RangeTest::Eq, + RangeMatch::subject_predicate(s, p), + RangeOptions::new().with_to_t(ledger.t()), + ) + .await + .expect("range"); + let mut out: Vec = flakes + .iter() + .filter(|f| f.op) + .map(|f| match &f.o { + FlakeValue::String(v) => v.clone(), + other => format!("{other:?}"), + }) + .collect(); + out.sort(); + out +} + +/// A ledger with a named graph registered under its own address, which a +/// TriG block can create. `USING
` with `GRAPH
` +/// templates: the templates write that graph, and a WHERE row witnesses a +/// template only when the graph the WHERE read is the graph the template +/// writes, compared as graph ids. The row's value is inserted into that +/// graph whichever graph the WHERE reads for the address, and the default +/// graph is left alone. +#[tokio::test] +async fn a_graph_named_by_the_ledger_address_gets_what_its_templates_write() { + for indexed in [false, true] { + let dir = tempfile::tempdir().expect("tempdir"); + let fluree = FlureeBuilder::file(dir.path().to_string_lossy().to_string()) + .build() + .expect("file fluree"); + let id = format!("legacyaddress{indexed}:main"); + let ledger = fluree.create_ledger(&id).await.expect("create"); + let trig = format!( + "@prefix ex: <{EX}> .\nex:s ex:p \"d\" .\nGRAPH <{id}> {{ ex:s ex:p \"l\" . }}\n" + ); + let mut ledger = fluree + .insert_turtle(ledger, &trig) + .await + .expect("seed") + .ledger; + assert!( + ledger + .snapshot + .graph_registry + .graph_id_for_iri(&id) + .is_some(), + "the fixture registers a graph under the ledger's address" + ); + if indexed { + drop(ledger); + support::rebuild_and_publish_index(&fluree, &id).await; + ledger = fluree.ledger(&id).await.expect("reload"); + } + let r = sparql_update( + &fluree, + ledger, + &format!( + "DELETE {{ GRAPH <{id}> {{ ?s ex:p ?o }} }} \ + INSERT {{ GRAPH <{id}> {{ ?s ex:p \"d\" }} }} \ + USING <{id}> WHERE {{ ?s ex:p ?o }}" + ), + ) + .await; + assert!( + graph_objects(&r.ledger, Some(&id)) + .await + .contains(&"d".to_string()), + "indexed={indexed}: {:?}", + graph_objects(&r.ledger, Some(&id)).await + ); + assert_eq!( + graph_objects(&r.ledger, None).await, + ["d"], + "indexed={indexed}" + ); + } +} + +// ============================================================================= +// A DELETE can name a stored ill-typed literal +// ============================================================================= + +/// A file-backed ledger holding the Turtle `ttl`, optionally indexed. +async fn seeded_turtle( + name: &str, + ttl: &str, + indexed: bool, +) -> (tempfile::TempDir, Fluree, LedgerState) { + let dir = tempfile::tempdir().expect("tempdir"); + let fluree = FlureeBuilder::file(dir.path().to_string_lossy().to_string()) + .build() + .expect("file fluree"); + let id = format!("it/delete-stored-{name}-{indexed}:main"); + let ledger = fluree.create_ledger(&id).await.expect("create"); + let ledger = fluree + .insert_turtle(ledger, ttl) + .await + .expect("seed") + .ledger; + if !indexed { + return (dir, fluree, ledger); + } + drop(ledger); + support::rebuild_and_publish_index(&fluree, &id).await; + let ledger = fluree.ledger(&id).await.expect("reload"); + (dir, fluree, ledger) +} + +/// A Turtle write keeps an ill-typed literal as its lexical form with the +/// declared datatype. A DELETE names that stored term: its template keeps the +/// lexical form as stored instead of refusing it, on SPARQL and JSON-LD. An +/// insert still refuses one. +#[tokio::test] +async fn a_delete_names_a_stored_ill_typed_literal() { + let ttl = format!( + "@prefix ex: <{EX}> .\n@prefix xsd: .\n\ + ex:s ex:n \"abc\"^^xsd:integer .\n" + ); + for indexed in [false, true] { + for surface in ["sparql", "jsonld"] { + let (_d, fluree, ledger) = + seeded_turtle(&format!("ill-typed-{surface}"), &ttl, indexed).await; + assert_eq!(facts(&fluree, &ledger).await, ["n=abc"], "precondition"); + let r = if surface == "sparql" { + sparql_update( + &fluree, + ledger, + "DELETE DATA { ex:s ex:n \"abc\"^^xsd:integer }", + ) + .await + } else { + jsonld_update( + &fluree, + ledger, + json!({"delete": {"@id": "ex:s", "ex:n": {"@value": "abc", "@type": "xsd:integer"}}}), + ) + .await + }; + assert_eq!(r.receipt.retract_count, 1, "{surface}, indexed={indexed}"); + assert!( + facts(&fluree, &r.ledger).await.is_empty(), + "{surface}, indexed={indexed}" + ); + } + } + + // An insert still refuses an ill-typed literal. + let text = format!( + "PREFIX ex: <{EX}>\nPREFIX xsd: \n\ + INSERT DATA {{ ex:s ex:n \"abc\"^^xsd:integer }}" + ); + let parsed = fluree_db_sparql::parse_sparql(&text); + let mut ns = fluree_db_transact::NamespaceRegistry::new(); + assert!( + fluree_db_transact::lower_sparql_update_ast( + &parsed.ast.expect("ast"), + &mut ns, + fluree_db_transact::TxnOpts::default(), + ) + .is_err(), + "INSERT DATA refuses an ill-typed literal" + ); +} diff --git a/fluree-db-api/tests/it_delete_witness_routing.rs b/fluree-db-api/tests/it_delete_witness_routing.rs new file mode 100644 index 0000000000..59f6fba722 --- /dev/null +++ b/fluree-db-api/tests/it_delete_witness_routing.rs @@ -0,0 +1,283 @@ +//! The witnessed-DELETE lane must actually fire, and stand down when the +//! ledger can hold a list position. +//! +//! A DELETE whose template a WHERE triple witnesses (`DELETE { ?s ex:p ?o } +//! WHERE { ?s ex:p ?o }`) retracts its rows as decoded, with no stored-fact +//! read, when neither the index nor novelty holds an `@list` position; on a +//! list-bearing ledger the rows read their slots for list positions. The +//! answers agree either way, so only the `delete_witnessed` routing stamp +//! can tell which lane ran. Both directions are asserted on the same ledger: +//! `MustFire` while it is list-free, `MustNotFire` after one `@list` lands. +//! +//! Its own `[[test]]` binary, like `it_upsert_skip_fires`: tracing's +//! callsite-interest cache is process-global, so a sibling test in a bundled +//! binary can leave the stamp's callsite disabled. + +#[path = "support/span_capture.rs"] +mod span_capture; + +use fluree_db_api::{Fluree, FlureeBuilder, LedgerState, TransactResult}; +use serde_json::{json, Value as JsonValue}; + +fn ctx() -> JsonValue { + json!({"ex": "http://example.org/ns/"}) +} + +fn witnessed_delete(subject: &str) -> JsonValue { + json!({ + "@context": ctx(), + "where": {"@id": subject, "ex:tag": "?o"}, + "delete": {"@id": subject, "ex:tag": "?o"} + }) +} + +#[tokio::test(flavor = "current_thread")] +async fn delete_witnessed_lane_fires_only_on_a_list_free_ledger() { + let (store, _guard) = span_capture::init_test_tracing(); + let site = fluree_db_transact::DELETE_WITNESSED_SITE; + let outcomes = |from: usize| -> Vec { + store.find_events("fast-path outcome")[from..] + .iter() + .filter(|e| e.fields.get("site").map(String::as_str) == Some(site)) + .filter_map(|e| e.fields.get("outcome").cloned()) + .collect() + }; + + let fluree = FlureeBuilder::memory().build_memory(); + let ledger = fluree + .create_ledger("it/delete-witness-routing:main") + .await + .expect("create"); + let ledger = fluree + .insert( + ledger, + &json!({ + "@context": ctx(), + "@graph": [ + {"@id": "ex:a", "ex:tag": "x"}, + {"@id": "ex:b", "ex:tag": "y"}, + {"@id": "ex:c", "ex:tag": "z"} + ] + }), + ) + .await + .expect("seed") + .ledger; + assert!(!ledger.novelty.has_list_meta); + + // MustFire: list-free. + let before = store.find_events("fast-path outcome").len(); + let r = fluree + .update(ledger, &witnessed_delete("ex:a")) + .await + .expect("witnessed delete"); + assert_eq!(r.receipt.retract_count, 1); + assert_eq!( + outcomes(before), + ["proceed"], + "a witnessed delete on a list-free ledger must take the zero-read lane" + ); + + // A DELETE DATA-shaped delete has no witness and stamps nothing. + let before = store.find_events("fast-path outcome").len(); + let r = fluree + .update( + r.ledger, + &json!({"@context": ctx(), "delete": {"@id": "ex:b", "ex:tag": "y"}}), + ) + .await + .expect("constant delete"); + assert_eq!(r.receipt.retract_count, 1); + assert!(outcomes(before).is_empty(), "no witness, no stamp"); + + // One @list on the same ledger turns the lane off. + let ledger = fluree + .insert( + r.ledger, + &json!({"@context": ctx(), "@id": "ex:l", "ex:items": {"@list": ["a", "b"]}}), + ) + .await + .expect("list") + .ledger; + assert!(ledger.novelty.has_list_meta); + + // MustNotFire: list-bearing. + let before = store.find_events("fast-path outcome").len(); + let r = fluree + .update(ledger, &witnessed_delete("ex:c")) + .await + .expect("witnessed delete, list-bearing"); + assert_eq!(r.receipt.retract_count, 1, "the delete still retracts"); + assert_eq!( + outcomes(before), + ["fallback:gate_declined"], + "a list-bearing ledger must read list positions" + ); +} + +/// An update on either surface. +enum Update { + JsonLd(JsonValue), + Sparql(&'static str), +} + +async fn apply(fluree: &Fluree, ledger: LedgerState, update: Update) -> TransactResult { + match update { + Update::JsonLd(txn) => fluree.update(ledger, &txn).await.expect("JSON-LD update"), + Update::Sparql(body) => { + let text = format!("PREFIX ex: \n{body}"); + let parsed = fluree_db_sparql::parse_sparql(&text); + assert!(!parsed.has_errors(), "{text}: {:?}", parsed.diagnostics); + let mut ns = fluree_db_transact::NamespaceRegistry::from_db(&ledger.snapshot); + let txn = fluree_db_transact::lower_sparql_update_ast( + &parsed.ast.expect("ast"), + &mut ns, + fluree_db_transact::TxnOpts::default(), + ) + .expect("lower"); + fluree + .stage_owned(ledger) + .txn(txn) + .execute() + .await + .expect("SPARQL update") + } + } +} + +/// A value bound by the sole triple of an OPTIONAL, or of one UNION branch, +/// is the decode of that triple's fact, so its row takes the witnessed lane: +/// the JSON-LD and SPARQL shapes, and Cypher `SET` and `DETACH DELETE`, +/// which read the old value through an OPTIONAL. A UNION whose other branch +/// binds the object from another predicate stays on the matched lane. +#[tokio::test(flavor = "current_thread")] +async fn optional_and_union_rows_take_the_witnessed_lane() { + let (store, _guard) = span_capture::init_test_tracing(); + let site = fluree_db_transact::DELETE_WITNESSED_SITE; + let outcomes = |from: usize| -> Vec { + store.find_events("fast-path outcome")[from..] + .iter() + .filter(|e| e.fields.get("site").map(String::as_str) == Some(site)) + .filter_map(|e| e.fields.get("outcome").cloned()) + .collect() + }; + + let fluree = FlureeBuilder::memory().build_memory(); + let ledger = fluree + .create_ledger("it/delete-witness-optional:main") + .await + .expect("create"); + let mut ledger = fluree + .insert( + ledger, + &json!({ + "@context": ctx(), + "@graph": [ + {"@id": "ex:a", "@type": "ex:Node", "ex:tag": "x"}, + {"@id": "ex:b", "@type": "ex:Node", "ex:tag": "y"}, + {"@id": "ex:c", "@type": "ex:Node", "ex:tag": "z"}, + {"@id": "ex:d", "@type": "ex:Node", "ex:tag": "w", "ex:other": "v"}, + {"@id": "ex:e", "@type": "ex:Node", "ex:tag": "u"}, + {"@id": "ex:f", "@type": "ex:Node", "ex:tag": "t"}, + {"@id": "ex:g", "@type": "ex:Node", "ex:tag": "s"}, + {"@id": "ex:h", "@type": "ex:Node", "ex:tag": "r", "ex:other": "q"} + ] + }), + ) + .await + .expect("seed") + .ledger; + + // (case, update, witnessed). Each retracts one fact either way; where + // the other UNION branch binds the object from another predicate, the + // row goes through the resolver. + let cases = [ + ( + "JSON-LD OPTIONAL", + Update::JsonLd(json!({ + "@context": ctx(), + "where": [{"@id": "ex:a", "@type": "ex:Node"}, + ["optional", {"@id": "ex:a", "ex:tag": "?o"}]], + "delete": {"@id": "ex:a", "ex:tag": "?o"} + })), + true, + ), + ( + "JSON-LD UNION", + Update::JsonLd(json!({ + "@context": ctx(), + "where": [["union", {"@id": "ex:b", "ex:tag": "?o"}, {"@id": "ex:b", "@type": "?t"}]], + "delete": {"@id": "ex:b", "ex:tag": "?o"} + })), + true, + ), + ( + "SPARQL OPTIONAL", + Update::Sparql( + "DELETE { ex:f ex:tag ?o } WHERE { ex:f a ex:Node OPTIONAL { ex:f ex:tag ?o } }", + ), + true, + ), + ( + "SPARQL UNION", + Update::Sparql( + "DELETE { ex:g ex:tag ?o } WHERE { { ex:g ex:tag ?o } UNION { ex:g a ?t } }", + ), + true, + ), + ( + "JSON-LD UNION binding the object twice", + Update::JsonLd(json!({ + "@context": ctx(), + "where": [["union", {"@id": "ex:d", "ex:tag": "?o"}, {"@id": "ex:d", "ex:other": "?o"}]], + "delete": {"@id": "ex:d", "ex:tag": "?o"} + })), + false, + ), + ( + "SPARQL UNION binding the object twice", + Update::Sparql( + "DELETE { ex:h ex:tag ?o } WHERE { { ex:h ex:tag ?o } UNION { ex:h ex:other ?o } }", + ), + false, + ), + ]; + let mut failures = Vec::new(); + for (what, update, witnessed) in cases { + let before = store.find_events("fast-path outcome").len(); + let r = apply(&fluree, ledger, update).await; + let stamps = outcomes(before); + let expected: &[&str] = if witnessed { &["proceed"] } else { &[] }; + if r.receipt.retract_count != 1 || stamps != expected { + failures.push(format!( + "{what}: {} retracted, stamps {stamps:?}", + r.receipt.retract_count + )); + } + ledger = r.ledger; + } + assert!(failures.is_empty(), "{failures:#?}"); + + for (what, stmt) in [ + ( + "Cypher SET", + "MATCH (n:`http://example.org/ns/Node`) WHERE n.`http://example.org/ns/tag` = 'z' \ + SET n.`http://example.org/ns/tag` = 'zz'", + ), + ( + "Cypher DETACH DELETE", + "MATCH (n:`http://example.org/ns/Node`) WHERE n.`http://example.org/ns/tag` = 'u' \ + DETACH DELETE n", + ), + ] { + let before = store.find_events("fast-path outcome").len(); + let r = fluree.transact_cypher(ledger, stmt).await.expect(what); + assert!(r.receipt.retract_count >= 1, "{what}"); + assert!( + outcomes(before).iter().any(|o| o == "proceed"), + "{what}: {:?}", + outcomes(before) + ); + ledger = r.ledger; + } +} diff --git a/fluree-db-api/tests/it_fast_stats_1391_regression.rs b/fluree-db-api/tests/it_fast_stats_1391_regression.rs index b798050f15..0a7403ad0c 100644 --- a/fluree-db-api/tests/it_fast_stats_1391_regression.rs +++ b/fluree-db-api/tests/it_fast_stats_1391_regression.rs @@ -110,27 +110,69 @@ fn novelty_doc() -> Value { }) } -/// The third commit, covering the *under*-count half of #1391. Novelty accepts -/// every retraction — `apply_commit`'s dedup gate short-circuits on -/// `flake.op &&` and never examines one — so a ground DELETE that matches -/// nothing still lands as a retraction flake and, under a blind delta log, -/// subtracts one from a count that never included it. -/// -/// * `ex:w2 ex:name "never-existed"` matches nothing: must charge zero; -/// * `ex:w1 ex:size 1` is a real base fact: must charge `-1`, taking -/// `ex:size`'s ledger-wide count to zero so the property drops out of -/// `apoc.meta.data` and `db.propertyKeys()` entirely. +/// The third commit, covering the *under*-count half of #1391: `ex:w1 +/// ex:size 1` is a real base fact and must charge `-1`, taking `ex:size`'s +/// ledger-wide count to zero so the property drops out of `apoc.meta.data` +/// and `db.propertyKeys()` entirely. fn deletion_doc() -> Value { json!({ "@context": {"ex": "http://example.org/ns/"}, "where": {}, "delete": [ - {"@id": "ex:w2", "ex:name": "never-existed"}, {"@id": "ex:w1", "ex:size": 1} ] }) } +/// The fourth commit: a retraction of `ex:w2 ex:name "never-existed"`, which +/// matches nothing and must charge zero. Novelty accepts every retraction — +/// `apply_commit`'s dedup gate short-circuits on `flake.op &&` and never +/// examines one — so under a blind delta log it subtracts one from a count +/// that never included it. +/// +/// A transaction no longer writes such a retraction (a DELETE of an absent +/// fact stages nothing), but commits written before that, and commits +/// replayed from elsewhere, still carry them. So the commit is built by hand +/// and staged the way a replayed commit is. +async fn commit_noop_retraction( + fluree: &Fluree, + ledger: fluree_db_api::LedgerState, + index_config: &IndexConfig, +) { + use fluree_db_core::{Flake, FlakeValue, Sid}; + let s = ledger + .snapshot + .encode_iri("http://example.org/ns/w2") + .expect("ex:w2"); + let p = ledger + .snapshot + .encode_iri("http://example.org/ns/name") + .expect("ex:name"); + let xsd_string = Sid::new(fluree_vocab::namespaces::XSD, "string"); + let t = ledger.t() + 1; + let noop = Flake::new( + s, + p, + FlakeValue::String("never-existed".into()), + xsd_string, + t, + false, + None, + ); + let ns = fluree_db_transact::NamespaceRegistry::from_db(&ledger.snapshot); + let view = fluree_db_transact::stage_flakes( + ledger, + vec![noop], + fluree_db_transact::StageOptions::new(), + ) + .await + .expect("stage the no-op retraction"); + fluree + .commit_staged(view, ns, index_config, CommitOpts::default()) + .await + .expect("commit the no-op retraction"); +} + /// `(label, property)` pairs the schema shims report, each paired with the /// SPARQL that counts the same facts by scanning. The hand-pinned number is /// the current-state truth; both the stats answer and the scan must equal it. @@ -275,6 +317,8 @@ async fn setup() -> (tempfile::TempDir, Fluree) { ) .await .expect("deletion update"); + let ledger = fluree.ledger(ALIAS).await.expect("reload after deletion"); + commit_noop_retraction(&fluree, ledger, &index_config).await; let ledger = fluree.ledger(ALIAS).await.expect("reload after deletion"); let retractions = ledger diff --git a/fluree-db-api/tests/it_import.rs b/fluree-db-api/tests/it_import.rs index b95e5136ee..f59551dfea 100644 --- a/fluree-db-api/tests/it_import.rs +++ b/fluree-db-api/tests/it_import.rs @@ -1689,8 +1689,8 @@ GRAPH { #[tokio::test] async fn import_trig_compact_named_graph_is_queryable() { // Issue #1278: the compact W3C TriG form ` { ... }` (no GRAPH keyword) - // must import through the bulk path too — `import_trig_commit` shares the - // same `parse_trig_phase1` parser as the upsert endpoint. + // must import through the bulk path too, which reads TriG with + // `parse_trig_phase1`. let db_dir = tempfile::tempdir().expect("db tmpdir"); let data_dir = tempfile::tempdir().expect("data tmpdir"); diff --git a/fluree-db-api/tests/it_join_batched_overlay.rs b/fluree-db-api/tests/it_join_batched_overlay.rs index ef9a52b36a..63b36e3499 100644 --- a/fluree-db-api/tests/it_join_batched_overlay.rs +++ b/fluree-db-api/tests/it_join_batched_overlay.rs @@ -483,8 +483,8 @@ async fn batched_object_join_merges_novelty() { .expect("reindex base"); // Novelty tail: b2's author retracted, b3 asserted, grace + b4 exist only - // in novelty, and b5's two-entry author @list is fully retracted (both - // flakes share (s, p, o) and differ only in o_i). + // in novelty, and one entry of b5's two-entry author @list is retracted + // (both flakes share (s, p, o) and differ only in o_i). let receipt = fluree .update( receipt.ledger, @@ -499,14 +499,15 @@ async fn batched_object_join_merges_novelty() { let receipt = fluree .update( receipt.ledger, + // A constant delete names one entry of a value held at several + // list positions; a WHERE-driven delete would name every one. &json!({ "@context": ctx(), - "where": {"@id": "ex:b5", "ex:authors": "?a"}, - "delete": {"@id": "ex:b5", "ex:authors": "?a"} + "delete": {"@id": "ex:b5", "ex:authors": {"@id": "ex:bob"}} }), ) .await - .expect("retract b5 author list"); + .expect("retract one b5 author entry"); let _receipt = fluree .insert( receipt.ledger, @@ -542,10 +543,9 @@ async fn batched_object_join_merges_novelty() { r"PREFIX ex: SELECT ?x WHERE { VALUES ?b { ex:bob ex:dave } ?x ex:authors ?b } ORDER BY ?x", - 1, // the delete's WHERE dedupes the repeated @list value to one - // binding, so exactly ONE of the two o_i entries is retracted: - // the fate check must discriminate the entries by o_i (a - // default-read o_i would either resurrect both or drop both) + 1, // exactly ONE of the two o_i entries is retracted: the fate + // check must discriminate the entries by o_i (a default-read o_i + // would either resurrect both or drop both) true, ), ( diff --git a/fluree-db-api/tests/it_named_graph_conflict_retractions.rs b/fluree-db-api/tests/it_named_graph_conflict_retractions.rs new file mode 100644 index 0000000000..280bc7d11a --- /dev/null +++ b/fluree-db-api/tests/it_named_graph_conflict_retractions.rs @@ -0,0 +1,195 @@ +//! Conflict retractions in a named graph read index-resident values. +//! +//! A take-source merge, a take-source revert and a take-branch rebase each +//! retract the losing side's current values on a conflicting `(subject, +//! predicate, graph)`. They read those values with a range read that +//! returns index-resident rows without their graph, then kept only the rows +//! whose graph equalled the key's. In a named graph that dropped every +//! indexed value, so the losing value survived next to the winning one +//! (novelty-resident values were read correctly). The read now goes through +//! the retraction resolver, which stamps the graph. + +use crate::support; +use fluree_db_api::{CommitRef, ConflictStrategy, FlureeBuilder, MergePreviewOpts}; +use serde_json::{json, Value as JsonValue}; + +const G1: &str = "http://example.org/g1"; + +fn set_value(v: &str) -> JsonValue { + json!({ + "@context": {"ex": "http://example.org/"}, + "graph": G1, + "where": {"@id": "ex:s", "ex:p": "?o"}, + "delete": {"@id": "ex:s", "ex:p": "?o"}, + "insert": {"@id": "ex:s", "ex:p": v} + }) +} + +async fn values(fluree: &support::MemoryFluree, ledger_id: &str) -> Vec { + let ledger = fluree.ledger(ledger_id).await.expect("load"); + let q = format!( + "SELECT ?o WHERE {{ GRAPH <{G1}> {{ ?o }} }}" + ); + let r = support::query_sparql(fluree, &ledger, &q) + .await + .expect("query"); + let mut out: Vec = r + .to_jsonld(&ledger.snapshot) + .expect("jsonld") + .as_array() + .expect("rows") + .iter() + .map(|row| { + let cell = row.as_array().and_then(|r| r.first()).unwrap_or(row); + cell.as_str() + .map(str::to_string) + .unwrap_or_else(|| cell.to_string()) + }) + .collect(); + out.sort(); + out +} + +async fn seed_graph(fluree: &support::MemoryFluree, ledger: fluree_db_api::LedgerState) { + fluree + .insert( + ledger, + &json!({ + "@context": {"ex": "http://example.org/"}, + "@graph": [{"@id": "ex:s", "@graph": G1, "ex:p": "orig"}] + }), + ) + .await + .expect("seed"); +} + +/// Take-source merge: the target's value is retracted whether it is in +/// novelty or in the index, and the preview shows it either way. +#[tokio::test] +async fn take_source_merge_replaces_the_targets_named_graph_value() { + for indexed in [false, true] { + let fluree = FlureeBuilder::memory().build_memory(); + let ledger = fluree.create_ledger("mydb").await.unwrap(); + seed_graph(&fluree, ledger).await; + fluree + .create_branch("mydb", "dev", None, None) + .await + .unwrap(); + let dev = fluree.ledger("mydb:dev").await.unwrap(); + fluree.update(dev, &set_value("frombranch")).await.unwrap(); + let main = fluree.ledger("mydb:main").await.unwrap(); + fluree.update(main, &set_value("frommain")).await.unwrap(); + if indexed { + support::rebuild_and_publish_index(&fluree, "mydb:main").await; + } + + let preview = fluree + .merge_preview_with( + "mydb", + "dev", + None, + MergePreviewOpts { + include_conflict_details: true, + conflict_strategy: ConflictStrategy::TakeSource, + ..MergePreviewOpts::default() + }, + ) + .await + .unwrap(); + let detail = preview.conflicts.details.first().expect("one conflict"); + assert_eq!( + detail.target_values.len(), + 1, + "indexed={indexed}: the preview must show the target's value" + ); + + fluree + .merge_branch("mydb", "dev", None, ConflictStrategy::TakeSource) + .await + .unwrap(); + assert_eq!( + values(&fluree, "mydb:main").await, + ["frombranch"], + "indexed={indexed}" + ); + } +} + +/// Take-source revert: reverting the commit that wrote the value also +/// retracts the value HEAD holds now. +#[tokio::test] +async fn take_source_revert_retracts_the_indexed_head_value() { + for indexed in [false, true] { + let fluree = FlureeBuilder::memory().build_memory(); + let ledger = fluree.create_ledger("mydb").await.unwrap(); + let r1 = fluree + .insert( + ledger, + &json!({ + "@context": {"ex": "http://example.org/"}, + "@graph": [{"@id": "ex:anchor", "ex:name": "anchor"}] + }), + ) + .await + .unwrap(); + let r2 = fluree + .insert( + r1.ledger, + &json!({ + "@context": {"ex": "http://example.org/"}, + "@graph": [{"@id": "ex:s", "@graph": G1, "ex:p": "v1"}] + }), + ) + .await + .unwrap(); + let v1_commit = r2.receipt.commit_id.clone(); + fluree.update(r2.ledger, &set_value("v2")).await.unwrap(); + if indexed { + support::rebuild_and_publish_index(&fluree, "mydb:main").await; + } + fluree + .revert_commits( + "mydb", + "main", + vec![CommitRef::Exact(v1_commit)], + ConflictStrategy::TakeSource, + ) + .await + .unwrap(); + assert!( + values(&fluree, "mydb:main").await.is_empty(), + "indexed={indexed}: the revert wins, so v2 goes too" + ); + } +} + +/// Take-branch rebase: the branch's value wins and the source's value, +/// index-resident on the source, is retracted. +#[tokio::test] +async fn take_branch_rebase_retracts_the_sources_indexed_value() { + for indexed in [false, true] { + let fluree = FlureeBuilder::memory().build_memory(); + let ledger = fluree.create_ledger("mydb").await.unwrap(); + seed_graph(&fluree, ledger).await; + fluree + .create_branch("mydb", "dev", None, None) + .await + .unwrap(); + let dev = fluree.ledger("mydb:dev").await.unwrap(); + fluree.update(dev, &set_value("dev-val")).await.unwrap(); + let main = fluree.ledger("mydb:main").await.unwrap(); + fluree.update(main, &set_value("main-val")).await.unwrap(); + if indexed { + support::rebuild_and_publish_index(&fluree, "mydb:main").await; + } + fluree + .rebase_branch("mydb", "dev", ConflictStrategy::TakeBranch) + .await + .unwrap(); + assert_eq!( + values(&fluree, "mydb:dev").await, + ["dev-val"], + "indexed={indexed}" + ); + } +} diff --git a/fluree-db-api/tests/it_rdf_text_parity.rs b/fluree-db-api/tests/it_rdf_text_parity.rs new file mode 100644 index 0000000000..381b81705c --- /dev/null +++ b/fluree-db-api/tests/it_rdf_text_parity.rs @@ -0,0 +1,1118 @@ +//! RDF text is parsed once, by the Turtle parser, on every write lane. +//! +//! Upsert read Turtle and TriG through a JSON-LD round trip: the text was +//! parsed into a graph, re-encoded as JSON-LD and expanded again. The round +//! trip refused IRIs whose scheme the compact-IRI guard took for an +//! undefined prefix (`tag:`, `kb:`; #1977), stored collections as unordered +//! values, and parsed literals more strictly than insert. Upsert now stages +//! from the parse itself, so it stores what insert stores. + +use crate::support::{genesis_ledger, rebuild_and_publish_index, MemoryFluree}; +use fluree_db_api::policy_builder::build_policy_context_from_opts; +use fluree_db_api::{ + CommitOpts, Fluree, FlureeBuilder, GovernanceOptions, GraphDb, GraphPayload, LedgerState, + PolicyContext, TrackingOptions, TransactionOptions, TxnOperation, +}; +use fluree_db_core::comparator::IndexType; +use fluree_db_core::{ + range_with_overlay, ContentStore as _, Flake, FlakeValue, RangeMatch, RangeOptions, RangeTest, + Sid, +}; +use fluree_db_transact::{GraphSel, TxnOpts}; +use serde_json::{json, Value as JsonValue}; + +const TAG: &str = "tag:example.org,2025:"; +const KB: &str = "kb:n#"; + +fn memory() -> MemoryFluree { + FlureeBuilder::memory().build_memory() +} + +// ============================================================================= +// Reading back what a lane stored +// ============================================================================= + +/// A node as written in the document: its IRI, or `_:label` for a blank +/// node, whatever skolem scope the lane minted it under +/// (`fdb--