flaky: it_property_path_batched lane-engagement asserts fail ~2% under parallel cargo test --test grp_query_reason #1697

Test infrastructure, not product code — but it's a false-fail that lands on whoever is next to run the gates, and it's the third instance of the same thread-local-set_default pattern (#1486's minmax thesis, and it_count_datatype_pin before it). Reproduced at 47e96b7b6 with the prebuilt grp_query_reason binary.

Measurement

Every failure was the same assert, and it was always an engagement assert, never a row-count assert:

thread 'it_property_path_batched::property_path_batched_wildcard_cypher' panicked at
fluree-db-api/tests/it_property_path_batched.rs:291:5:
fused wildcard run should engage the raw-id lane on a clean HEAD

So the rows the engine produced were correct on every single run; only the evidence that the fast lane ran went missing. (The #1486 battery recorded the same failure in the sibling test property_path_batched_lane_clean_and_novelty, whose identical assert is at :125. Same file, same mechanism — I'd treat them as one item.)

Mechanism

The lane's event is emitted unconditionally on entry, before any decline logic:

// fluree-db-query/src/property_path.rs:1846-1851
tracing::debug!(batched = batched.len(), fallback = fallback.len(), forward,
                "property path raw-id expand");

batched may be empty and the event still fires. So a missing event means either the lane was never entered, or the event was emitted and not captured. The serial and isolated runs above rule out the first: nothing about the lane's decision depends on what other tests are doing — the view is a per-test in-memory ledger — and it engaged 85/85 times when no sibling module was running concurrently.

That leaves capture. span_capture::init_test_tracing (fluree-db-api/tests/support/span_capture.rs:323-329) installs its subscriber with tracing::subscriber::set_default, which is thread-local, while tracing's callsite-interest cache is process-global. grp_query_reason bundles nine modules and none of the other eight uses span_capture at all — but three of them (it_query_property_path, it_query_shortest_path, it_query_seq_path_count_repro) drive property paths and therefore hit this exact callsite with no capturing subscriber on their thread. Meanwhile it_property_path_batched creates and drops a fresh Dispatch per query inside its loop. I'm not going to claim I've pinned the exact instruction — but shared process-global tracing state plus per-query dispatcher churn plus concurrent siblings hitting the same callsite is the same interference class #1486 described, and the empirical signature (only under cross-module parallelism, only the absence-of-evidence asserts) matches it.

Worth noting the CI angle: nextest's process-per-test model hides this completely, which is why it's still latent. Bare cargo test is what most of us reach for first.

Suggested direction

Give the file its own test binary, the way #1486 did for it_minmax_fast_path_fired and main had already done for it_count_datatype_pin — a [[test]] entry in fluree-db-api/Cargo.toml and remove the #[path] include at fluree-db-api/tests/grp_query_reason.rs:4-5. That makes the capture deterministic without weakening the assert, which is the part worth keeping: the whole point of these tests is proving the fast lane actually ran.

If we'd rather not add another binary, the alternative is a process-global subscriber installed once for the bundle with per-test filtering — but that's a bigger change and it re-introduces the cross-test coupling in a different place, so I'd lean to the standalone bin.

Notes

• Unrelated to test: it_query_rdfs_class_repro flaky under concurrent nextest; bulk-import staging not cleaned #1271 (also a grp_query_reason flake, but bulk-import staging contention with a different signature).
• Same family as flaky: binary-index reload race in values_bound_type_after_delete_incremental_index_timeline under parallel grp_misc (dict branch: too small for header) #1489 and the it_minmax_fast_path_fired move in test: standalone it_minmax bin + aws-gate smoke_minio (vbench follow-ups) #1486, though those are storage-reload and absence-assert cases respectively.
• Reproducing it: RUST_MIN_STACK isn't needed here, just loop the full binary — for i in $(seq 1 100); do cargo test -p fluree-db-api --test grp_query_reason || break; done.