perf(local): skip full GC for CPU reranker providers

[Sanderhoff-alt]

flowchart LR
    A[LocalST or FlashRank inference] --> B{device type}
    B -->|cpu| C[heap trim]
    B -->|cuda / xpu / mps| D[gc.collect]
    B -->|unknown| D
    D --> C
    C -->|GPU only| E[empty_gpu_cache]
    C -->|CPU| F[return scores]
    E --> F

Loading

Decision

Path	Before	After	Rationale
LocalST CPU reranker	gc.collect() + heap trim	heap trim	Avoid a process-wide full cyclic-GC scan on every hot-path batch.
FlashRank CPU reranker	gc.collect() + heap trim	heap trim	The same shared cleanup helper applies; measurements show the same benefit.
CUDA/XPU/MPS local reranker	gc.collect() + heap trim + GPU cache cleanup	unchanged	Preserve the existing accelerator memory-release policy.
Unknown device	full cleanup	unchanged	Conservative fallback when the device cannot be identified.

This change gates the single explicit gc.collect() call in release_local_inference_memory() on device_type == "cpu". Both LocalSTCrossEncoder and FlashRankCrossEncoder call this helper from their post-inference finally blocks, so the policy applies consistently to both CPU providers. The change does not disable Python's automatic cyclic GC, change reference counting, remove heap trimming, or alter model inference semantics.

Problem

gc.collect() without a generation argument performs a synchronous full cyclic-GC scan for the entire Python process. Local reranker inference runs in a dedicated worker thread, but the collection is process-wide and adds latency to every completed inference. CPU inference already releases most short-lived tensors, ONNX buffers, and tokenization containers through normal reference counting, while Python's cyclic GC continues to run on its normal allocation thresholds.

The remaining heap operation, malloc_trim on Linux or malloc_zone_pressure_relief on macOS, addresses native allocator high-water marks rather than Python object cycles. It is therefore kept separate from the full-GC decision and remains on every CPU reranker call.

Benchmark Results

The following fixed-size CPU benchmarks use median wall-clock milliseconds per inference call. Lower is better. Each table uses a cached model and the same three cleanup variants: inference only, full gc.collect(), and the normal heap-trim path.

LocalSTCrossEncoder

Pairs	Inference only	Inference + gc.collect()	Inference + heap trim
1	3.6 ms	76.3 ms	3.4 ms
8	7.0 ms	80.7 ms	6.9 ms
32	15.5 ms	89.2 ms	16.2 ms
128	60.6 ms	135.0 ms	60.6 ms

The full-GC penalty is approximately 73-74 ms per LocalST call across the tested candidate sizes. Heap trim is within measurement noise of inference-only execution on this macOS environment.

FlashRankCrossEncoder

Pairs	Inference only	Inference + gc.collect()	Inference + heap trim
1	2.0 ms	19.0 ms	2.0 ms
8	15.8 ms	32.4 ms	14.0 ms
32	52.3 ms	69.9 ms	50.7 ms
128	207.7 ms	226.7 ms	210.3 ms

FlashRank's full-GC penalty is approximately 17-19 ms per call. The absolute penalty is smaller than LocalST because ONNX inference is slower, but it is still a synchronous and unnecessary cost on every CPU request. Heap trim remains close to the control and is retained.

FlashRank RSS stress check

Both variants below kept heap trim enabled and differed only in whether they additionally called full GC. Values are macOS ru_maxrss high-water measurements in MB from separate fresh processes.

Cleanup policy	Round 0	Round 10	Round 20	Round 30	Tracked objects
CPU skip full GC + heap trim	314.1	992.4	993.1	993.4	133,529 throughout
CPU full GC + heap trim	313.7	992.0	992.5	992.8	133,342 throughout

The high-water mark reaches a stable plateau in both processes; full GC does not provide a measurable RSS advantage for this workload. The tracked-object count remains constant within each process, which does not indicate accumulating Python cycles.

Benchmark Methodology

Environment: Apple Silicon arm64 MacBook Pro, macOS 26.6.2, Python 3.11.14, PyTorch 2.10.0, SentenceTransformers 5.2.0, and ONNX Runtime via FlashRank. LocalST used cross-encoder/ms-marco-MiniLM-L-6-v2; FlashRank used ms-marco-MiniLM-L-12-v2. Both ran on CPU with batch size 32 and no GPU work. The model artifacts were cached before timing; the FlashRank zip was obtained from a Hugging Face-compatible mirror after the direct Hugging Face endpoint timed out, and download time was excluded.

For each provider and candidate size (1, 8, 32, and 128 pairs), the harness used one stable query and deterministic short document strings. Each cleanup variant performed three warmup calls followed by eight measured calls. Timing used time.perf_counter() around the synchronous provider call, and the reported value is the median of the eight samples.

The cleanup variants isolate the cost by replacing the post-inference cleanup callback with: no cleanup (inference-only control), gc.collect() only, or the normal local cleanup helper (heap trim on CPU). The deployed CPU path corresponds to inference plus heap trim, not the inference-only control.

A separate LocalST variable-length workload ran 120 rounds and compared no explicit full GC against full GC after every round. The no-GC run had a 43.63 ms median and 50.13 ms mean; the per-round-GC run had a 125.35 ms median and 133.69 ms mean. RSS after 120 rounds was approximately 829-830 MB in both cases.

The LocalST stress check was extended to 240 rounds. With explicit GC disabled, RSS moved from 554.9 MB at round 0 to 829.6 MB at round 120 and then plateaued at 830.6 MB by round 240. In a separate run with Python automatic GC disabled, RSS moved from 556.0 MB to 831.9 MB, tracked objects changed from 488,397 to 488,465, and a final manual gc.collect() returned 0. The initial RSS increase is consistent with model warmup and allocator high-water behavior; the later plateau and zero collected objects provide no evidence of an accumulating cyclic-reference leak.

Design Logic

The proposed behavior follows the ownership of each cleanup mechanism:

• Reference counting handles the normal lifetime of temporary CPU tensors, ONNX buffers, and Python containers when the inference call returns.
• Automatic cyclic GC remains enabled and can reclaim cycles when its normal thresholds are reached; a cycle is not expected to require a full scan after every request.
• Heap trim remains unconditional because it targets freed native pages retained by the C allocator, which is independent of Python cyclic GC.
• GPU paths retain the previous ordering: collect Python wrappers, trim native heap, then empty the backend allocator pool.

gc.collect() is process-wide even though it is called from a reranker worker thread. Removing it from the CPU path therefore reduces contention for all threads in the API process, not just the worker that issued the call.

Scope and Compatibility

This is intentionally a narrow policy change. There are no new configuration variables, no provider API changes, no model changes, and no changes to GPU cleanup behavior. LocalSTCrossEncoder on CPU and CPU-only FlashRankCrossEncoder both take the optimized path; CUDA, XPU, and opt-in MPS retain full cleanup. CPU embedding behavior was already guarded separately and is unchanged by this patch.

Validation

• uv run pytest -q hindsight-api-slim/tests/test_local_device.py hindsight-api-slim/tests/test_local_cross_encoder.py
• Result: 43 passed in 8.45 seconds; only existing dependency deprecation warnings were emitted.
• ./scripts/hooks/lint.sh passed during implementation.

The focused tests verify that CPU cleanup still trims the heap without calling gc.collect(), while GPU cleanup still calls GC, heap trim, and the matching backend cache release.

Operational Follow-up

After rollout, monitor reranker wall time/P95-P99 latency, process RSS, and container OOM events for local CPU deployments. If a future provider introduces a reproducible cyclic-reference regression, full GC can be reintroduced for that provider or changed to a periodic or threshold-triggered policy without changing the device cleanup interface.

[strix-security]

Strix Security Review

Warning

This pull request has 1 commit after the last Strix review (c990ed2). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

No security issues found.

_{Updated for c990ed2.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}