Repository navigation
Test harness: span capture can miss events under cargo test, and two benches time their TempDir drop #2070
Description
Two harness problems that make our gates read better than they are, both deferred from the PRs that ran into them. Neither is product code, but the first is the kind of false-fail that lands on whoever runs the gates next, and the fast-path suites' routing stamps (MustFire / MustNotFire) only mean something if span capture sees every event.
- Span capture can miss an event.
tracingcaches a callsite's interest on its first hit, and while at most one capturing subscriber is registered it asks only the hitting thread's subscriber. So a test thread with no capture that hits a callsite first caches "never" until the next subscriber registers, and a routing check on another thread can then see no stamp for one query. A never-dropped second registrar influree-db-api/tests/support/span_capture.rs(interest "sometimes",enabledfalse) would close it for every test binary. It shows up in bundledcargo testruns (it_annotation_filter_pushdown,it_limit_stops_work), not undernextest, which runs each test in its own process; A class of test flake CI cannot see by construction: overlay tests fail undercargo testsame-process concurrency, andnextesthides it #1895 has more on why CI can't see this class. I think this is also the mechanism behind flaky:it_property_path_batchedlane-engagement asserts fail ~2% under parallelcargo test --test grp_query_reason#1697's missing lane-engagement events, so it's worth re-running flaky:it_property_path_batchedlane-engagement asserts fail ~2% under parallelcargo test --test grp_query_reason#1697's measurement once this lands. - Two benches time their
TempDirdrop.fluree-db-api/benches/transact_filtered_delete.rsandfluree-db-api/benches/novelty_replay.rsdrop their file-backedTempDirinside the timed routine, so the unlink of the directory lands in the measurement.transact_upsert_replace.rs, which fix(transact): retractions name stored facts; RDF text is parsed once #2008 added, doesn't; these two should follow. Fixing them moves their numbers, so both should change together, with a baseline reset.
From #2008 (Follow-ups: the racy span-capture tests; the two benches) and #2010 (Not changed here: the span-capture harness).
Activity
aaj3f commentedon Oct 13, 2026
Three more sightings of the first item, all local
cargo testruns, each passing on every rerun. Each failure is a routing-stamp check whose span capture saw no events at all:it_jsonld_join_batched_lane::jsonld_join_takes_batched_subject_probe_like_sparql(grp_query):left: [], once in a full-binary run; 5/5 alone and 3/3 full reruns pass.it_policy_predicate_fast_lanes::class_rule_declines_covered_predicate(grp_policy):stamps: []; 140/140 on three reruns.it_query_optional_encoded_bindings::cypher_optional_match_reverse_edge_over_encoded_values(grp_query):stamps: [], in a run filtered to that one module, where nothing touches the fast-path kill switch.The third one makes the kill switch (
it_differential_fastpathflippingset_fast_paths_disabled) a weaker suspect than it looks, and the failures cross test binaries. The likelier common cause is the capture harness itself:support::span_capture::init_test_tracinginstalls a thread-local subscriber (tracing::subscriber::set_default), while plaincargo testruns a binary's tests on concurrent threads that share one process and tracing's per-callsite interest cache. CI runscargo nextest(one process per test), so it can't happen there; it's local-only.A cheap discriminator: loop
grp_queryundercargo test -- --test-threads=1versus the default thread count, with the kill-switch test filtered out.