refactor(storage): shared byte handles from get; the disk cache as a store wrapper

[bplatz]

Partially addresses #2029: stages 1–3. Stage 4 (encryption above the cache) is not done; see Departures from #2029.

Readers used to decide how to read by asking the store around the trait: resolve_local_path, resolve_cached_bytes, resolve_local_bytes, is_remote and permits_plaintext_cache. The disk cache sat outside every store and needed those answers, so a cache_dir was threaded through the binary index, dictionary, pack and vector readers, the arena loaders, incremental resolution and the GC walks. After this PR, readers ask a store for bytes and get a shared handle. The cache is a store wrapper that is applied once, where the store stack is assembled.

75 files, +3,218 / −3,473, in one commit per stage.

1. ContentStore::get returns a shared ContentBytes handle

ContentBytes derefs to [u8] and has three variants. Every variant is Arc-backed, so cloning one never copies the bytes.

• Owned(Arc<Vec<u8>>): heap bytes.
• Shared(Arc<[u8]>): bytes the store already holds, such as memory storage's.
• Mapped(Arc<Mmap>): a file mapping. Not compiled on wasm.

cache_weight() keeps the existing leaflet-cache discount for mappings and charges heap bytes in full. Callers that need a Vec use into_vec(), which moves the bytes when the handle is the only owner.

Incremental builds keep the handles: the prefetched leaves and sidecars stay ContentBytes, and update_branch takes any B: AsRef<[u8]> + Send from its fetch closures. A mapped leaf is never copied into the heap, and handing it to the copy-on-write update is a refcount bump.

2. One synchronous get_local replaces the local-read hooks

get_local(id) -> Result<Option<ContentBytes>>, on both StorageRead and ContentStore, replaces resolve_local_path, resolve_local_bytes and ContentStore::resolve_cached_bytes:

• Some(bytes) when this machine already has the object;
• None when it must be fetched;
• an error when a local copy exists but cannot be read.

How each storage serves it:

• File storage: opens the blob, falling back to the WAL replay it already did on a miss. A zero-length file is answered as None, since it is failed-write debris. The rest is read into the heap at or below 64 KiB and mapped above that (the threshold the pack reader used). ContentBytes::from_file and open are unsafe fn: a mapping is sound only while the file is never rewritten, and every writer lands files by renaming a staged file into place.
• Memory storage: shares its Arc<[u8]>.
• Encrypted storage: decrypts its inner storage's local copy.
• Tiered storage: routes the call like every other read.

StorageRead::resolve_cached_bytes survives only as the wasm residency hook, which the bridge consults first.

The leaf, dictionary-leaf, forward-pack, vector-shard and sidecar readers try get_local first and fetch on None. Arenas and branches read through get.

• Three leaf handle types (FullBlobLeafHandle, SharedBlobLeafHandle, MmapLeafHandle) become one.
• The dictionary reader classifies a leaf when it is looked up, rather than stating every leaf when the reader is built.
• A leaf served by get_local is looked up outside the leaflet cache's single-flight loader. Otherwise every memory-storage open would pay a block_in_place. Shared bytes are not inserted into the leaflet cache, since the store already holds them.
• The exception is storage that encrypts at rest, where get_local decrypts the whole leaf. A new hint, ContentStore::costly_local_reads, says so; the bridge answers it from encryption_admin. On such a store the leaf is read through the loader: once per leaf however many cursors open it, inside a blocking region.

3. Fetch leaves whole from stores without ranged reads

A new ContentStore::supports_ranged_reads() hint defaults to false. On a store without ranged reads every range request is a whole-object fetch, plus a decrypt for encrypted storage. A leaf on such a store is now fetched whole on its first open, and the fetched bytes go into the leaflet cache. Stores with ranged reads keep the existing policy: range reads first, and a whole fetch from the leaf's second open. BranchedContentStore reports ranged reads only when every ancestor supports them.

4. The disk cache becomes CachedContentStore

StorageBackend::with_disk_cache(cache_dir) is the one place that asks whether a storage is remote:

• Remote plaintext storage: wrapped with the cache.
• Remote storage that decrypts on read: gets coalesced fetches only, and nothing is written to disk.
• Local storage: left alone, and the cache directory is never created.
• Permanent stores: coalesced fetches only. with_disk_cache can't ask one whether its bytes may sit on disk in plaintext, so a store opts in with StorageBackend::cached_permanent. IPFS, which serves bytes as published, does.
• Routed backends: the rule is applied to each mount.

Applying it twice changes nothing. Every Fluree and LedgerManager constructor applies it over LedgerManagerConfig::cache_dir (or the default directory), and so does the indexer start.

CachedContentStore behaviour:

• Reads: the inner store's local tier first, then the cache entry, then a coalesced fetch that writes the entry. It never copies what the inner store serves locally.
• What is cached: index artifacts only: roots, branches, leaves, dictionaries, sidecars, stats sketches, spatial artifacts and garbage records. Commits, transactions, config and graph-source snapshots are fetched and coalesced but never written. Otherwise a reindex or replay of a long history would push hot leaves out of a cache that evicts in write order, and put transaction data on local disk.
• Range reads: served from the entry when one exists, sized from the entry, so mid..u64::MAX reads to the end.
• Empty entries: read as misses, and the next fetch replaces them. A crash can leave one, since entries are renamed into place without an fsync.
• Releases: reach the inner store first, then evict the entry, including for batch releases. The process-wide eviction walk is gone.
• Entries: keyed by CID and shared by every store using the directory.
• Coalescing: a waiter takes the leader's result without running its own read. So a fetch is shared only among one backend's readers of one namespace. A child branch's miss must not fail its parent's concurrent read, and a reader holding the wrong key must not receive another backend's plaintext. Single-flight logic is now its own Flights<K> type, and DiskArtifactCache::coalesced_fetch, which Iceberg uses, keeps its path keys on top of it.

Two hints replace the explicit seeding and prewarm calls. Both are no-ops by default:

• keep_local: bytes the indexer just wrote, for reads about to follow.
• prefetch: warm the local tier ahead of a synchronous read. Used by the indexer and by BinaryIndexStore::prefetch_* on the query path.

put does not write through, so a full rebuild does not churn the cache.

Removed:

• ContentStore::is_remote and permits_plaintext_cache. The storage still answers both, and assembly reads them once.
• uses_disk_cache, needs_disk_copy, seed_disk_cache, fetch_cached_bytes* and evict_cached_cid.
• IndexerConfig::artifact_cache_dir() and CleanGarbageConfig::artifact_cache_dir.
• Every cache_dir parameter on the readers, incremental resolution and the GC walks.

Ledger verification reads through StorageBackend::without_disk_cache(), because a cached copy would hide an object the storage has lost.

Departures from #2029

• Stage 4 (encryption above the cache) is not done. It is a separate design, not a relocation of existing code:

  • put(ciphertext) derives a different CID, and put_with_id(plaintext_cid, ciphertext) fails the bridge's verification.
  • Key rotation rewrites the envelope in place, so a CID-keyed ciphertext cache would serve an old-key envelope after the old key is retired.
  • Equal plaintext CIDs under different key providers would collide in one cache directory.

  Encrypted remote stores remain uncached, as today. Storage: return shared handles from get and move the disk cache into a store wrapper #2029 stays open for this stage.

• is_remote and permits_plaintext_cache stay on StorageRead, still required with no default, but are read only at assembly, not by readers.

• No write-through on put; seeding is explicit, through keep_local.

• The range-first and promotion policy stays in the leaf reader. The wrapper serves range reads from its entries.

Operator-visible changes

• FLUREE_DICT_PACK_MMAP_MIN_BYTES is now FLUREE_MMAP_MIN_BYTES, and covers every blob file storage serves, not only dictionary packs. A deployment that set the old name loses its setting.
• Cache entries are named by CID. Entries written under the old {digest}.{ext} names are left until budget eviction removes them.
• Fluree::new, from_backend, with_indexing_mode and a directly constructed LedgerManager cache remote reads, in the default directory or LedgerManagerConfig::cache_dir. Before this PR they did so through each reader's own cache directory.
• The indexer's disk cache moves. It was <data_dir>/binary_artifact_cache whenever IndexerConfig::data_dir was set; it is now the shared cache in LedgerManagerConfig::cache_dir (default $TMPDIR/fluree_binary_cache). A deployment that sets data_dir but not cache_dir now caches index artifacts in the temp directory. Nothing opens or evicts an existing binary_artifact_cache directory any more, so it can be deleted.
• FLUREE_MMAP_MIN_BYTES is documented in docs/operations/configuration.md.

API changes for code that implements storage traits

• ContentStore::get returns ContentBytes.
• ContentStore and StorageRead gain get_local, which replaces resolve_local_path and resolve_local_bytes.
• ContentStore gains supports_ranged_reads, costly_local_reads, prefetch and keep_local, each with a default.
• A wrapper must forward all five new methods. Arc<dyn ContentStore>, BranchedContentStore, MeteredContentStore and CachedContentStore each have a test that it does.
• update_branch and update_branch_streaming take a byte type B: AsRef<[u8]> + Send from their fetch closures; closures returning Vec<u8> still work.
• StorageBackend::cached_permanent(store, cache_dir) is new.
• Fluree::build_index(ledger_id) builds a ledger's index in-process with the API's indexer setup: the full-text config provider and the plaintext-staging warning. fluree index calls it. The warning moved out of the indexer, and the CLI, which built its own IndexerConfig, had stopped reaching it.

Tests

Each lane change and wrapper rule is pinned by a test. I checked each pin by reverting the code it covers, 55 reverts in all, and every one turned its test red. The main pins:

• core:
  • ContentBytes threshold and weight tests;
  • FileStorage::get_local: zero-length, small/large, unreadable;
  • get_local at every candidate address on memory storage;
  • hint forwarding for Arc<dyn> and BranchedContentStore, including the parent fallback and the all-ancestors rule for ranged reads;
  • CachedContentStore: served from its entry, never copies inner-local bytes, release before evict, keep_local/prefetch, coalescing in both modes, open-ended range reads from an entry, empty entries as misses, only index artifacts written.
• binary-index:
  • leaves from a store without ranged reads are fetched whole once, with and without a directory-first open;
  • resident leaves, dictionary leaves, packs and vector shards are not fetched or copied;
  • directory-only opens of local leaves;
  • dictionary leaves are classified only on lookup;
  • concurrent opens of a leaf on a store whose local reads decrypt read it once.
• indexer: MeteredContentStore forwards every hint.
• core, after review: a_permanent_store_is_disk_cached_only_by_opt_in. Giving Permanent a disk cache by default turns it red.
• api, after review: build_index_warns_when_encrypted_storage_stages_in_temp: encrypted storage warns and plain storage doesn't. Removing the warning from build_index turns it red. The encryption tests no longer check binary_artifact_cache, which nothing writes; the shared cache they already check covers the indexer.
• crypto: get_local serves plaintext; an encrypted remote backend writes nothing to the cache directory.
• api:
  • it_disk_cache_locality:
    • local and memory storage write nothing, and never create the directory;
    • remote builds seed the cache;
    • encrypted storage writes nothing;
    • a wrong-key reader fails even while the right key's fetch is in flight;
    • a child branch's miss does not fail a concurrent parent read, with and without a disk cache;
    • verify reports a commit the storage lost while the cache still holds it.
  • Every Fluree constructor wraps remote storage; LedgerManager::new caches in its configured directory.
  • Tiered storage routes get_local.

Run on this branch, rebased on 745736a04:

• fmt, workspace clippy, and the wasm32 clippy gate: clean.

Suite	Passed
core (native)	966
binary-index	410 (413 with residency)
indexer	437
crypto	29
Iceberg	271
api lib	841
api grp_index	126
api grp_misc	318
api grp_ledger	249
api grp_transact	250
api it_sql_pushdown_lane	19

The fetch-count pins (it_rebuild_fetch_counts, it_reindex_pre_build_perf, it_memory_storage_reads) pass unchanged.

After the review fixes: core 999 (all features), indexer 439, api lib 839, grp_index 127, grp_misc 320.

Adjacent paths checked

• Other callers that treat get returning NotFound as a signal (commit export, merge, pack, sidecar prefetch) only read what exists. A cached copy has the same bytes, so they are unaffected. Verify was the one caller asking whether storage itself holds an object, and it now reads storage directly.
• GC prev-index chain walks read through the cache, as they did before through artifact_cache_dir. A cached copy of a root that storage has already released lets the walk run past where it should end. This is documented on PrevIndexChainWalk::next_entry, and the sweep compensates with an existence check. It is not changed here.

Not verified

• Merge gate, not yet met: the A/B benchmarks. The gate is ≤ 1.02× of the base.
  • Reads: at parity on DBLP and BSBM in the 10-11 A/B (comment below).
  • Writes: that A/B measured BSBM Update at 0.94× throughput, with index builds 4–6% slower. The review traced part of it to fetch_noting_local copying each local leaf twice, which is now fixed. The Update A/B has not been re-run since.
• Windows: file storage now maps every blob over 64 KiB, not only leaves and packs. Windows can't delete or rename over a mapped file, so GC or a ledger drop on a Windows file store may fail where a reader still holds a mapping. No CI job runs Rust storage on Windows.
• The annotation_hydration/scan re-measure for annotation_hydration/scan is 11–18% slower than #2018's merge base over memory storage #2030.
• S3 itself. Remote lanes are covered through MemoryStorage::simulating_remote and fetch counts, not S3 timings.
• The Python package's pytest suite.

[bplatz]

A/B benchmark: this PR vs next

• a = next at 9398f8e1a, the base this branch merged.
• b = this PR at a66519271.

Both read the same imported store, since the on-disk format is unchanged. Boxes were EC2 m7a.4xlarge, with runs rotated between variants. "Remote" is MinIO on a separate box with 12 ms added to each request, which stands in for S3's per-GET latency.

Reads: no regression

DBLP SPARQLoscope, 105 queries, 3 repeats each. Ratios are b ÷ a over per-query medians, so below 1 means this PR is faster.

	Geomean latency	Queries ≥ 100 ms
File storage	1.005	0.997
Remote, empty disk cache	0.990	1.000
Remote, after restart over the filled disk cache	1.005	1.001
Remote, hot	0.966	0.981

• All 315 saved outputs (105 queries × 3 runs) are byte-identical between a and b.
• Storage requests are the same: about 56,300 per cold pass and 38,300 after a restart, in every repeat.

BSBM 100M, file storage, 2 repeats each. Query mixes per hour, so above 1 means this PR is faster.

	b ÷ a
Explore, 1 client	1.032
Explore, 8 clients	1.033
BI, 1 client	1.007
BI, 8 clients	1.008

Writes: incremental index builds look ~4–6% slower

BSBM 100M Update, file storage, background indexing on, 3 repeats each.

• With the leaflet cache at 22,000 MB, both variants peaked at 54–59 GB on a 64 GB box, and b's third run was killed for running out of memory at 63 GB. So close to the limit, throughput says little about either variant.

• Rerun with the cache at 8,000 MB, both peaked at 36–50 GB with no kills:

  	a	b
  Update query mixes/hour (mean of 3)	8,465	7,934 (0.94×)
  Median steady-state index build (90–100 commits)	6.49 s	6.74 s (+4%)

  With the 22,000 MB cache, the same build measure was 7.07 s for a and 7.51 s for b (+6%).

The samples are small and Update throughput is noisy (b ranged from 7,162 to 9,262). Even so, every way of slicing the build times points the same direction. Not profiled yet. One candidate is fetch_noting_local, which now maps each leaf through get_local and copies it with into_vec, where it used to read the file in one call.

Memory pressure, and #2068

Both variants run near the box's limit under Update because next still warms every leaflet of every rewritten leaf, with all columns and no concurrency bound. #2068 (stacked on #2053, targeting main) warms only the leaflets readers had cached, on a bounded pool. In its own BSBM Update measurements it cut peak RSS from about 56 GB to about 40 GB and dictionary re-reads from about 220 GB to about 25 GB. I expect much of the gap above to be this pressure: the index build competes with warming for memory and for reads of the dictionary.

The build-time difference persisted with the 8,000 MB cache, so the rerun doesn't settle it. Once #2068 reaches next, I'll rerun the Update comparison to separate the two.

Also seen, not caused by this PR

On SIGINT, the server panics after main returns ("A Tokio 1.x context was found, but it is being shutdown"). It happens on both a and b, in every remote run.

[aaj3f]

@bplatz this is the design #2029 asked for, and I'm fully on board, including deferring stage 4. Your three reasons (CID derivation, in-place key rotation, cross-key collisions) are exactly why it isn't a relocation. I spent most of the review on plaintext. I drove get/get_local/get_range/prefetch/keep_local over encrypted remote storage as a managed backend, a branched child and a routed mount, and nothing reached disk; dropping the gate in with_disk_cache turns your crypto test red. Two headline notes:

(1) The plaintext invariant now lives in one chokepoint, which is the point. But that chokepoint can't see a Permanent store and caches it unconditionally (inline). Narrower alternative: make Permanent opt in to the cache.

(2) This overlaps #2048 head-on. It adds the same whole-leaf-on-first-open mechanism for stores without ranged reads, and removes the four things #2048 builds on (fetch_cached_bytes_cid, uses_disk_cache, cache_dir, SharedLeafBytes; conflicts in eight regions of binary_index_store.rs). I'd land #2048 on main and merge main into next before this merges, so the Whole policy becomes one more trigger of fetch_whole_leaf here, in a branch with its own tests, rather than being re-derived in a routine merge. There's also a one-line conflict with #2051 in rebuild.rs. Otherwise good to merge once the write-side A/B meets the PR's own gate.

Full review below:

---

✅ Approving to unblock you — with 1 thing that needs to be addressed before this merges.

This is a big, careful refactor, @bplatz. The departures section, the operator-visible list and the "55 reverts, all red" testing note made it far easier to review than its size suggests. The read side checks out: your A/B shows no regression with byte-identical outputs, and I found no correctness behavior lost among the removed hooks. A read-only pass traced all ~300 base call sites of resolve_local_path/resolve_cached_bytes/resolve_local_bytes/is_remote/permits_plaintext_cache/uses_disk_cache/fetch_cached_bytes*/evict_cached_cid/artifact_cache_dir to their replacements, and I re-checked the flagged ones.

Must address before merge:

1. The incremental-build regression against the PR's own ≤ 1.02× gate (incremental.rs:129). fetch_noting_local now maps each local leaf and copies it with into_vec, and the CoW closure clones it again. I measured that step at +25% per 721 KiB leaf versus base's fs::read. Holding ContentBytes in leaf_map/sidecar_map, and letting update_branch's fetch closures return it, removes both copies (−30% versus base in the same bench). Worth doing before the Update A/B re-run.

Should address:

• The plaintext-staging warning for encrypted storage is gone from the CLI's fluree index and direct indexer callers (rebuild.rs:120). It now fires only from reindex and background indexing start.

Optional, in the inline notes: Permanent stores are always disk-cached, since with_disk_cache can't ask them; and two operator-visible changes aren't in the body's list (the indexer cache moving off <data_dir>/binary_artifact_cache, orphaning it; FLUREE_MMAP_MIN_BYTES missing from the config docs).

One question with no anchor: Windows is a dist target, and mapping now covers every FileStorage blob over 64 KiB, not only leaves and packs. A mapped file on Windows can't be deleted or renamed over, so GC or drop on a Windows file store may now hit more "in use" failures. No CI job runs Rust storage on Windows, so I couldn't check it.

Adherence to repo commitments:

• Patterns/abstractions: ✔ This is the class fix for per-reader cache decisions (#2018's local copies, #1916's decrypted artifacts, base's process-wide fetch_uncached flights): one assembly chokepoint and a wrapper readers can't bypass. Two stragglers are noted inline (Permanent, the staging warning).
• Performance (speed first, memory second): ✖ Reads are at parity per your A/B. CRITICAL until fixed: Update builds −4–6% against the PR's ≤ 1.02× gate, with a measured +25% per-leaf mechanism in fetch_noting_local and a fix inline. Memory: more artifact kinds are now mapped rather than heap, which is a win; evicted-but-mapped entries pin disk blocks only as long as readers hold them, same as leaves at base.
• Deployment targets: ⚠️ Shared code. Embedded in solo: no compile impact, Lambda's cache stays in /tmp, and encrypted S3 gets coalescing only. The standalone's old index/binary_artifact_cache is orphaned on a pin bump. wasm32 is a no-op wrapper and green. Windows mapping semantics are unexercised (question above). No fence violation found.
• Testing: ✔ The new pins run, and the ones I mutated have teeth. Dropping the plaintext gate turned an_encrypted_remote_backend_writes_nothing_to_the_disk_cache red. Dropping the namespace from the flight key turned a_child_branch_miss_does_not_fail_a_concurrent_parent_read red. My own matrix (encrypted × managed/branched/routed × every read and hint entry point) wrote nothing.
• Conventions: ✔ One commit per stage with thorough bodies; clippy and fmt green; storage-traits, content-id and encryption design docs updated. Config-doc gap noted inline.

Verified locally at branch HEAD (a66519271):

• PR tests by name: crypto lib (29), core disk_cache/storage/content_bytes (179), binary-index lib (367), indexer fuel (6), api grp_index incl. all seven it_disk_cache_locality tests (11).
• Two mutation checks, red and restored.
• Scratch tests: the encrypted-composition matrix and the Permanent probe; the fetch_noting_local copy bench.
• git merge-tree against origin/next (clean), #2048 and #2051.

Approving now so you can merge without waiting on another pass from me — just be sure the fetch_noting_local copy is gone and the Update A/B is back inside your 1.02× gate before you do.

[aaj3f]

🔴 Must address before merge: the incremental-build regression in your A/B has a measurable cause here, and a small fix that removes two copies

Your 2026-10-11 comment shows Update builds at 0.94× throughput and +4–6% build time, against this PR's own merge gate of ≤ 1.02× of main. You named fetch_noting_local as a candidate, and it holds up when measured. For a local leaf over 64 KiB, get_local maps the file, into_vec copies the mapping, and the munmap follows. Base did one fs::read. Then, as on base, the CoW's fetch closure .cloned()s that Vec again (:303-308), because update_branch takes Fn(&ContentId) -> io::Result<Vec<u8>> (run_index/build/incremental_branch.rs:112-124).

I timed it with a dev-fast scratch bench: 64 × 721 KiB files in page cache, every 4 KiB of the result touched, best of 3, macOS aarch64. Base fs::read + clone: 45–47 µs per leaf. This PR's map + into_vec + clone: 56–58 µs (+25%). With 8 threads: 11.2–11.6 → 14.4 µs. Linux's 4 KiB pages mean 4× the faults per mapped leaf, so I'd expect no smaller a gap on the EC2 box.

The fix that more than recovers it is to stop converting at all. Return ContentBytes from fetch_noting_local, hold it in leaf_map/sidecar_map, and let update_branch's fetch closures return ContentBytes (or a generic B: AsRef<[u8]>). The .cloned() becomes an Arc bump, and the prefetched set sits in page cache instead of heap. Same bench: 32 µs single-threaded (−30% vs base), parity at 8 threads. Illustrative:

async fn fetch_noting_local(
    content_store: &dyn ContentStore,
    id: &ContentId,
) -> fluree_db_core::Result<(ContentBytes, bool)> {
    if let Some(bytes) = content_store.get_local(id)? {
        return Ok((bytes, true));
    }
    Ok((content_store.get(id).await?, false))
}
// leaf_map: HashMap<ContentId, ContentBytes>; update_branch<F, G> with
// F: Fn(&ContentId) -> io::Result<ContentBytes>, G: … -> io::Result<Option<ContentBytes>>

I didn't profile a whole build, so I can't say this is all of the 4–6% (the warming pressure you mention from #2068 may be the rest). But it's PR-sized, and it should be in before the Update A/B re-run that decides whether the gate is met.

[bplatz]

Addressed in 5af2f24

[aaj3f]

🟠 Should address: the plaintext-staging warning for encrypted storage no longer fires on the CLI's fluree index or any direct indexer caller

Before this PR, the warning lived here, so every rebuild passed through it: data_dir unset over a store that refuses plaintext caching meant "index build staging defaults to the system temp directory". It moved to warn_if_staging_plaintext in the API (fluree-db-api/src/lib.rs:1451), which is called only from reindex (admin.rs:2264) and background-indexing start (lib.rs:3142). fluree index (fluree-db-cli/src/commands/index.rs:46-57) builds IndexerConfig::default(), so data_dir is None, and goes straight to build_index_for_ledger. On an encrypted store (the CLI has fluree encryption …), it now stages plaintext runs and leaves under $TMPDIR/fluree-index without a word. None of the commit bodies mention dropping it there.

The smallest fix is to call warn_if_staging_plaintext from the CLI's index_ledger too. A sturdier one is to keep the warning in the indexer, where every build path passes, by carrying "this store refuses plaintext" into IndexerConfig from the assembly that already knows it. Either way it's a few lines.

[bplatz]

Addressed in c0be36b

[aaj3f]

🟡 Optional: with_disk_cache can't ask a Permanent store whether it may be cached in plaintext, so it always caches it

This is more of a design note than a bug today. With permits_plaintext_cache/is_remote gone from ContentStore, this arm has no way to ask, and the doc assumes every Permanent store is IPFS. At base the reader asked the ContentStore, and StorageContentStore forwarded the storage's answer. So an encrypted store handed over as StorageBackend::Permanent(Arc::new(StorageContentStore::new(Arc::new(EncryptedStorage::new(…)), …))) was never cached. At this head, a scratch probe shows that same construction writes both a leaf and a root to the cache directory in plaintext.

In-repo only IPFS builds Permanent (fluree-db-api/src/lib.rs:2848), and I didn't find an embedder that does, so this is latent. But Permanent is a public variant, and this is now the one place the invariant lives. A narrow fix: leave Permanent uncached here and have the IPFS constructor opt in (e.g. StorageBackend::cached_permanent(store, dir)), so "a store that refuses plaintext never gets a disk" holds for every public construction. Minor and non-blocking, but if you agree it's right I'd rather see it folded in now than lost in the backlog.

[bplatz]

Addressed in 4c86d3e

[aaj3f]

🟡 Optional: two operator-visible changes the body's list doesn't include

1. The indexer's disk cache moves. At base, IndexerConfig::artifact_cache_dir() put it at <data_dir>/binary_artifact_cache whenever data_dir was set. Here the indexer's backend gets this wrapper with LedgerManagerConfig::cache_dir (default $TMPDIR/fluree_binary_cache). A deployment that sets data_dir but not cache_dir moves its indexer cache from a persistent path to the temp dir. Every deployment that had one leaves the old binary_artifact_cache directory orphaned: nothing opens it, so the budget never evicts it.
2. FLUREE_MMAP_MIN_BYTES (the rename the body does call out) isn't in docs/operations/configuration.md's table. perf(index): fewer requests for remote leaf reads; whole-leaf fetch by default #2048 adds FLUREE_REMOTE_LEAF_FETCH there.

A line for each under "Operator-visible changes", plus the docs row, would cover it. Stale binary_artifact_cache assertions remain in it_storage_encrypted.rs:325-385; they're vacuous now and could go in the same pass.

[bplatz]

Addressed in f68eef7 and c0be36b