[test] Isolate XQSuite test files, run CoreTests and XQuery3Tests in parallel

[duncdrum]

Follow-up to the JUnit 5 migration, which is merged (#6790, #6791). Part of #6037.

Summary

#6791 added parallel XQSuite files (@XQSuite(parallel = true)) but almost no suite could use it: test files share one database and collide on collection names and global state. This PR makes the files independent, enables parallel runs where that is safe, fixes the product bugs and reporting gaps the work turned up, and records what still blocks the rest.

"Isolated" means 6-12 parallel runs at 2, 4 and 8 files at a time are green. Parallelism is only the tool used to find the collisions.

Needs a closer look (changes under src/main): items 2 (XSLT catalog), 7, 8, 10, 11 (Lucene), 12 (three new public system: functions) and 15 (EXPath registry). Items 5 and 6 change the test reports, including test names. Items 2, 5 and 6 do not depend on the isolation work and can move into their own PRs.

What changed

Reports

1. [test] Publish the run time of each XQSuite file; one line per suite names the slowest.
2. [test] Failure reports show the expected and actual value (built with Jupiter's AssertionFailureBuilder, cut at 4000 characters). Before, surefire dropped them.
3. [test] Test names lead with their file (ft-query-field.xqm: issue5431-...) and carry the source line so an IDE can jump to the test. Reported names of XQSuite tests change, so CI history and flaky-test tracking restart.

Test engine (the first two commits; also open as #6823 and #6824, they drop out of this PR when one of those merges first)
19. [test] The engine's own fixture suites (the nested hang and failing-file classes of its tests) are skipped unless the test that sets them up asks for them (@XQSuite(fixture = true), configuration parameter exist.xqsuite.fixtures). A -Dtest wildcard that also matches nested classes, or an IDE package scan, ran them as ordinary suites and reported two failures for each of the hang fixtures after 5 minutes.
20. [bugfix] The wait for the threads of abandoned files is bounded by one grace period in all. It passed max(0, remaining) to Thread.join, and join(0) waits forever, so with two or more files stuck in a lock the run never ended (observed on #6775: XMLDBTests at 8 files at a time, still hung after almost two hours). Test: SuiteRunAwaitTerminationTest, three threads that ignore interruption; times out with the old formula.

Isolation
3. [test] Unique collection names per file, one commit per suite (CoreTests, XQuery3Tests, LogicalOpTests, SystemTests, OptimizerTests, RangeTests, FileTests, XmlDiffTests). Each file declares its collection name once, as a variable of the file; the rules for parallel suites are in the Javadoc of XQSuite#parallel().
12. [feature] Query-scoped trace functions system:enable-query-tracing, system:query-trace, system:clear-query-trace, used by the %test:stats handling. The database-wide trace made stats tests see other files' calls. The old functions are unchanged. Touches Profiler, QueryTrace, SystemModule, xqsuite.xql. Tests: QueryTraceTest (two overlapping queries) and two XQSuite guards for a stats test that throws.
17. [test] RangeTests queries are scoped to their own collection (part of the RangeTests commit): the six context-less range:field-eq calls of range.xql and the 17 range:index-keys-for-field calls of conditions.xql and prefixed-conditions.xql (without a context that function reads every document of the database). range.xql and conditions.xql also create a bystander collection with the same index and other data, so an unscoped query fails every time. On a loaded Windows CI runner one call found the copy of the same data stored by optimizer.xql (update-delete failed once; green in 30+ local runs). With the bystander present and without the scoping, 3 tests of range.xql and 5 of conditions.xql fail.
18. [test] FunDeepEqualPerformanceTest: the absolute timing caps become relative checks (fastest of 5 runs after 2 warm-ups). The 1500 ms root-mismatch test failed on earlier heads of this PR; it timed the construction of two in-memory trees (1019 to 1053 ms on three successful CI runs, longer than the equal-trees query) and could not detect a lost short-circuit. Now: stored equal documents at most twice a plain pass over them (about 0.29 locally), root mismatch on stored documents at most a tenth of the full comparison (1.3 ms against 155 ms), in-memory equal trees at most 5 times the construction alone. Checked by mutation: with the streaming path disabled the stored guard fails (3171 ms against 521 ms), with the root short-circuit removed the mismatch test fails (110.9 ms against 160.3 ms); the old caps passed both. The ratios on the CI runners are not measured yet.
4, 13, 14, 16, 21. [test] Parallel files enabled for CoreTests, XQuery3Tests (12 of 12 runs green each), RangeTests (24 of 24), OptimizerTests (12 of 12), ExpathRepoTests (12 of 12) and XMLDBTests (12 of 12 at 2, 4 and 8 files at a time, 4 reps each, 45 tests; it stayed serial because of the collection copy deadlock fixed in #6775, and without that fix the same setup deadlocked in 2 of 3 runs, so the suite also guards the lock order). NumericOpTests was already on.

Product fixes found on the way
2. [bugfix] XSLT imports: try the system catalog before the database resolver. The resolver fetched absolute http URLs with no connect timeout first; transform/catalog.xql waited 150 s on a non-routable address. XQuery3Tests 184 s -> 8 s (thread dump at 53 s).
7. [bugfix] Lucene ft:binary-field: read the value from the reader the search ran on, not a second snapshot (and compare with the segment size, not the live count). Was the flaky issue5431-sorted-by-binary-field (about 1 run in 5 at 8 files at a time, 0 of 40 after). Test: LuceneBinaryFieldLookupTest.
8. [bugfix] Lucene index scan (util:index-keys): fresh doc values iterators per term; they only move forward. Failed with an AssertionError on segments where only some documents have a node id. Test: LuceneIndexScanSparseDocValuesTest.
9. [test] Lucene tests no longer depend on score order or other files' documents; index-keys-by-qname tests use their own collection.
10. [bugfix] Lucene: analyze a document with the analyzers of the thread that adds it (FieldAnalyzerWrapper). One shared per-field registry let two threads swap each other's analyzer. Test: FieldAnalyzerWrapperTest (7). The interleaving was never observed directly; the evidence is a file pair that failed 16 of 74 runs before and 0 of 20 after.
11. [bugfix] Lucene ft:facets: keep the taxonomy reader alive as long as the facets can be read (AlreadyClosedException after a concurrent write and refresh). Held by a Cleaner, so an old reader stays open until a garbage collection. Test: LuceneFacetsReaderLifecycleTest.
15. [bugfix] EXPath package registry: serialize installs and removals. pkg-java 2.1.1 locks its package map only around parts of an install; choosing the directory, adding the version and the in-place rewrite of packages.xml / packages.txt run outside the lock, so concurrent calls read a half-written file (Error transforming the file: .../packages.xml) or could overwrite each other. ExistRepository now has installPackage / removePackage under a write lock (Deployment, PackageService, repo:install, repo:remove use them); a package that is not a local file is copied to a temporary file first so a slow download does not hold the lock; listPackages() returns a copy under a read lock because the library's own list is a live view that throws ConcurrentModificationException. Test: ExistRepositoryConcurrencyTest (5): concurrent install/remove failed 2 of 2 before and passes after; forced listing-versus-install interleaving; a stalled download does not block other installs or listings (fails when the copy is disabled); a failed download keeps its NotFoundException (pins existing behaviour). Limits: the Packages objects in a copy are still the library's; two installs of the same package run one after the other and the second replaces the first. The cleaner fix is in expath/expath-pkg-java.

Results

• Parallel and green: see items 4, 13, 14, 16, 21.
• Re-probed 2026-10-06 at 2, 4 and 8 files at a time, 2 runs each (6 per suite; 12 for OptimizerTests and ExpathRepoTests): every suite except the two blocked ones below is green in every run, with the same test count each time.
  • Parallel (7): CoreTests (70 files), XQuery3Tests (101), NumericOpTests (12), RangeTests (12), OptimizerTests (2), ExpathRepoTests (3), XMLDBTests (7; probed 2026-10-10 after the rebase, 12 runs).
  • Serial, isolated, 2 or more files (16): exist-core DateTests, IndexingTests, LogicalOpTests, MapTests, NumberTests, SecurityManagerTests, SystemTests, UtilTests, ValidationTests, XQSuiteTests; extensions LuceneIndexingTests, VectorSearchTests, NgramTests, CompressionTests, FileTests, XmlDiffTests. They run 0.6-7 s each, so parallel adds flake risk for no gain.
  • Serial, one file (14): nothing runs alongside them, so their runs say nothing about collisions: ArrayTests, InspectTests, XIncludeTests (13-14 s), and in extensions RestXqTests and XQDocTests (all tests skipped), IndexingTests (integration), FtAttrContextTests, FtMatchTests, LuceneAnalyzersTests, ReindexSecondStoreTest, SortTests, CacheTests, HttpClientXqueryTests, ImageTests.
• CI, ubuntu unit job, "Maven Unit Tests" step (push / PR run, measured before the reporting and Lucene commits): develop 1163 / 966 s; [refactor] Replace XSuite with a JUnit Platform engine, remove JUnit 4, move test support to the test-jar #6791 1139 / 917 s; this PR 884 / 630 s. Runner noise between runs of the same code is 200-250 s, so only the difference of about 300 s means anything; almost all of it is XQuery3Tests (297-310 s on develop, 18-32 s here). The longest classes stay Java tests (DeadlockIT, CollectionLocksTest), so no module-level speed-up is claimed.

Still serial

Suite	Cause	Status
LuceneTests	Defects found by parallel runs (items 7, 8, 10, 11) and the %test:stats files (item 12) are fixed. Rare failures remain at 4 and 8 files at a time: issue3977-reindex-doc-updates-not-adds (expected 1 2, got 1 1), some issue3042 / issue3043 sort tests, and now and then a cascade in ft-facets.xqm where one document's data stays wrong for the rest of the file. #6740 and #6756 were tried on top: no effect.	Open, root cause not found; tracked in #6800 (the XMLDBTests row of that issue is resolved by #6775 and item 21).

Not addressed here: #4455 (facet counts of consecutive ft:query calls on the same nodes add up; its test stays %test:pending). Related: #1752 (does not reproduce), #4538.

Test plan

• Rebased onto develop 198dabc0f3 (includes [refactor] Migrate tests from JUnit 4 to JUnit 5 #6790, [refactor] Replace XSuite with a JUnit Platform engine, remove JUnit 4, move test support to the test-jar #6791, [bugfix] omit newline after doctype with indent=no #5370 and [bugfix] Compare the full key when inserting into QNamePool #6795); getDocTypeIndentNo is a Jupiter test. Reactor test-compile; QNameTest, QNamePoolTest, NamePoolTest, SerializationTest (47, 0 failures); XQuery3Tests (1090, 0 failures, 3 skipped); RestoreAppsTest (6), PackageServiceJsonTest (4), ExpathRepoTests (9)
• On the final history: CoreTests 981 (3 of 3 runs at 8 files at a time), XQuery3Tests 1090, OptimizerTests 60, XQSuiteTestEngineTest 13, RangeTests 411, FileTests 54, ExpathRepoTests 9, ExistRepositoryConcurrencyTest 5 and the new Lucene tests, all green; after every cleanup step the tree was identical to the tested tree
• Each new guard test fails without its fix and passes with it (the registry stalled-download test checked by disabling the copy)
• Serial runs of every consumer of the query-scoped functions green: OptimizerTests 60, QuantifiedMatchOptimizerTest 12, XQSuiteTests 93, XQSuiteTestEngineTest 13, CoreTests 981, LuceneTests 679, RangeTests 411
• Each suite: serial run unchanged, 6 to 12 parallel runs green (above)
• Rebased 2026-10-10 onto develop c68693484d (includes [bugfix] Lock the destination Collection first when moving or copying #6775): reactor test-compile of all modules, then org.exist.test.xqsuite.* (all engine tests, 34), XMLDBTests 45, MoveCopyLockOrderTest 7, WebDavMoveCopyLockOrderTest 5, all green; XMLDBTests parallel 12 of 12
• CI on the current head 8ade722d3a (not run yet; the previous head d32ab7b80b was green on all jobs, the Windows integration job of an earlier head had failed RangeTests update-delete, fixed by item 17)

🤖 Generated with Claude Code

[duncdrum]

The timing failure on one of the unit runs is interesting. It looks like the kind of flake that surfaces once a bunch of test timings have changed. So in a way its expected, and will likely disappear on rerunning. Will dig into this more. I m at roughly 19/38 testuites that go through the whole lets isolate you dance.

[github-actions]

📊 XQTS result comparison

Comparison of this run against develop.

Metric	develop	this run	Change
➖ Passed	28,868 (90.74%)	28,868 (90.74%)	0 (0.00 pp)
➖ Failures	1,546	1,546	0
➖ Errors	134	134	0
➖ Skipped	1,267	1,267	0
🧪 Total tests	31,815	31,815	0

Relative to develop: 0 newly passing, 0 newly failing, 0 new errors, 0 newly skipped — counting only tests recorded in both runs whose outcome changed.

_{Runtime: 460.5s (+66.35s vs develop).}