Repository navigation
perf: remove the serial hot spots, and make a run reproducible from its config - #486
Merged
Merged
Conversation
… telemetry Profiling a 250k-peptide DIA run on a 120-core box showed ~90 % of wall time does not scale with threads; frame assembly (70 % of wall) spent ~85 % of its time on the main thread, dominated by the per-frame TDF writer. - tdf.py: dedup of (scan, tof) cells via a packed uint64 key + 1-D np.unique instead of np.unique(axis=0) + lexsort (10-15x faster, byte-identical output); keep analysis.tdf_bin open for the whole run instead of open/append/close per frame (34k syscalls, expensive on NFS). Handle is closed in write_frame_meta_data() and re-opened transparently if needed. - utility.py: python_list_to_json_string rounds the whole array once instead of a NumPy scalar round per element (called once per peptide/ion in four stages); output byte-identical, float32 inputs keep float32 rounding. - assemble_frames.py: per-batch timers for build / noise / write, logged and returned. - simulator.py: StageTracker records wall, CPU seconds (all threads), avg cores and peak RSS per stage, logs each transition, prints a stage table and writes timings.json next to synthetic_data.db. Sub-laps inside the FASTA stage (digest+rt_filter, sample_peptides). - tests: parity tests against the previous implementations, handle lifecycle, tracker output. Smoke run (5.7k frames, 2k peptides, 16 threads): assembly 54 s -> 38 s, total 156 s -> 138 s; .d reads back with imspy.
The Bruker SDK's tims_index_to_mz is not safe to call concurrently on one handle. rustdf handled that by dropping to a single thread whenever use_bruker_sdk is set, which is the default: get_pasef_fragments and get_preprocessed_pasef_fragments ignored num_threads entirely in SDK mode. The formula converter evaluates the calibration curves the .d file itself carries and is Sync. Measured against the SDK on a real reference over 73k peaks: worst |dmz| 3.2e-11 Th, worst 6.7e-8 ppm, mobility within one ULP, 54% of peaks bit-identical. That is orders of magnitude below any search tolerance, and below the 4e-5 Th run-to-run drift concurrent SDK calls were producing. So parallel regions now read through that converter instead of serialising: - TimsLazyLoder / TimsInMemoryLoader gain get_frame_with_converter, taking the converter explicitly rather than using self.index_converter. - TimsDataLoader gains sdk_free_converter() and raw_data_path(). - The two DDA PASEF paths and the DIA reference-noise overlay use it, and fall back to one thread only when the file carries no usable calibration tables. Adds a seeded overlay_reference_noise on TimsDatasetDIA (rayon, per-frame StdRng derived from seed and frame id) plus its connector and Python bindings, and generate_random_sample_with_rng on TimsFrame.
… overlay proteins_to_peptides_batched builds one sage indexed database per 2000 proteins instead of one per protein. generate_indexed_database sets up sage's parallel index build, so calling it 20000 times burnt ~3000 core-s on a 120-core box for a stage that ran faster with 4 threads. Peptide sets per protein are unchanged: sage attaches every protein a peptide occurs in. The DIA noise job now calls the batched, seeded overlay_reference_noise when the connector provides it, passing sample_seed, and keeps the serial per-frame loop as the fallback. Smoke A/B, 2000 peptides, 5692-frame DIA reference, 16 threads, identical config: wall 156 s released -> 138 s -> 119 s; frame assembly 54 -> 37.6 -> 21.6 s, of which the noise overlay is 18.9 -> 2.5 s. Output reads back with imspy; peak counts differ by the reseeded noise sampler. Tests cover batched-vs-per-protein digest parity with and without decoys, overlay determinism across 1/4/32 threads with the SDK enabled (the case that was flaky before the rustdf fix), and SDK vs formula converter parity.
The round-2 work and the rustdf SDK fix change behaviour and performance, so they need version numbers of their own: the overnight timing grid ran against a different code state that also called itself 0.4.3, which would have made the manuscript's version reference ambiguous. imspy-core moves too because the Python wrapper for the batched overlay lives there; without it the noise job silently falls back to the serial per-frame loop. imspy-simulation now requires >=0.4.3 of both so that cannot happen.
Profiling the TDF writer per step over 700 real frames (3.07M peaks) put 56 % of its time in the Bruker SDK's mz->tof and 1/K0->scan calls, 31 % in the interleave, and only 5 % in zstd — so the phase named "convert+compress+write" was never compression-bound, and tuning the zstd level would have been wasted. The conversions ran one frame at a time for the same reason the noise overlay did: they went through the SDK, which is not safe to call concurrently. In this direction the SDK-free converter is not merely close but exact — over 1,702,413 values, mz->tof and 1/K0->scan agree with the SDK bit for bit, because the outputs are integers and the last-bit float drift never crosses a rounding boundary. TDFWriter.write_frames now hands a whole batch to rustdf, which converts, dedups (scan, tof), interleaves and compresses across a rayon pool and returns the payloads plus the statistics the Frames row needs. Only the append and the running byte offset stay sequential. Passing numpy arrays straight through matters: an earlier parallel interleave existed but took Python lists, and the .tolist() marshalling alone was 21 % of writer time, which ate the gain. Output is byte-identical to the per-frame path, 700/700 frames at 1, 4 and 16 threads, and a test pins that. Getting there caught a real bug: zstd::encode_all omits the decompressed size from the frame header, and the simple API every existing .d was written with then refuses to read the frame. Uses zstd::bulk::compress, which writes it. Writer throughput on 700 frames: 1.36 s per-frame Python, 0.32 s batched at one thread, 0.056 s at 16 (24x). End-to-end smoke, 16 threads: the write phase falls 11.8 -> 2.5 s and frame assembly 21.6 -> 10.0 s. Also removes the commented-out compress_frames/write_frames skeleton this replaces; it was shelved as "needs debugging" and had omitted the dedup, which would have corrupted output since mz->tof is not injective.
…ly exact The exactness claim in the previous two commits rested on a single fixture. It does not generalise, and the code's own doc comment said so: BrukerFormulaConverter is bit-exact for MzCalibration ModelType 1, but ModelType 2 omits the proprietary C8..C14 fine correction and lands ~2 ppm away. Measured on the thunder blank, which is ModelType 2 and is a reference file the timing grid uses: feeding it arbitrary m/z — what the writer actually receives — 11.5 % of values land in a neighbouring TOF bin. That changes the (scan, tof) dedup and so the bytes written. The earlier test missed it twice over: its fixture is ModelType 1, and it fed back m/z the SDK itself had produced, which sit at bin centres and so are the case least likely to cross a rounding boundary. Mobility is exact under both models; only the m/z axis is affected. sdk_free_converter() now returns the converter only when is_sdk_exact() holds: every frame resolves to a real calibration row — no silent fallback to the default for NULL T1/T2, a missing Frames row, or an unsupported row — and every m/z row it uses is ModelType 1. Otherwise callers stay on the SDK, serially. Verified on real data: the ModelType-2 thunder blank is refused, the ModelType-1 dia blank accepted. Also hardened here: - build_compressed_frames validates that each frame's mz/mobility/intensity have equal length. A ragged triplet used to index out of bounds inside Rust; it now returns an error, as does a thread-pool or zstd failure instead of unwrapping. - The Python fallback in write_frames no longer swallows every exception. Only the explicit UNAVAILABLE capability signal falls back to the per-frame path; anything else propagates rather than quietly writing a different file. Tests: 193 Python passed / 15 skipped, 136 Rust passed.
The batched writer and the converter exactness gate change both behaviour and output timing, so they need their own version numbers: the 0.4.4 timing grid already ran against a different code state, and the manuscript cites a version.
The existing test compared compressed payloads in isolation. This writes the same frames through both paths and compares what lands on disk: the Frames table including the TimsId byte offsets, which depend on every preceding frame's compressed length, and the decompressed content of every frame. Running it on 600 frames turned up the one real nuance. The files are equivalent but not byte-identical: 4 bytes in 8.6 MB differ, in 2 frames of 600, entirely inside the zstd stream. Those blobs decompress to identical payloads, the compressed lengths match, and the Frames table is identical in all 18 columns. The cause is that the Python zstd module and the Rust zstd crate link different libzstd builds and can choose a different, equally valid encoding at the same cost. Reading both files back frame by frame — through the Bruker SDK and through the SDK-free reader — gives 600/600 identical frames over 2.55M peaks in mz, tof, scan and intensity. So the contract the test pins is the decompressed data and the table, not the byte encoding.
Two runs of an identical config with the same sample_seed shared only 11 of ~1690 peptides. Same configuration, effectively unrelated datasets. The 1.4 % peak-count difference that surfaced this was the mild symptom; the datasets underneath had almost nothing in common. Two independent causes, both upstream of anything seeded: 1. The global NumPy RNG was never seeded. sample_seed reached exactly two places — the pandas down-sample in simulator.py and (since 0.4.4) the reference-noise overlay. Everything deciding *what* gets simulated used bare np.random.*: peptide selection and abundance in simulate_peptides, the EMG retention-time shape, ion-mobility and scan variance, phosphosite choice, the decoy draw in digest_fasta. 16 call sites in jobs/ alone. 2. Both digest paths returned a `set` of peptide strings, and the sampler draws from it with np.random.choice. Set iteration order depends on Python's per-process string hash seed, so the draw differed run to run even with the RNG seeded — verified directly: same seed, three processes, three different selections. Both digests now return sorted lists. With both fixes every intermediate table is bit-identical between two runs: proteins, peptides, ions, fragment_ions, frames, scans. That is the ground truth the benchmarking compares against, so benchmarks are now reproducible from a config. Known remaining gap: the written .d still differs by ~0.002 % of peaks (570 of 29 M, and 37 counts of summed intensity). That is the m/z jitter — mscore's add_mz_noise_uniform/normal use rand::thread_rng at ~28 call sites across mscore and rustdf/src/sim. A shifted m/z occasionally crosses a TOF bin boundary and changes the dedup. Fixing it means threading a seeded per-spectrum RNG through those call sites, as generate_random_sample_with_rng already does for the overlay. Note this changes which peptides a given config selects: datasets generated before this cannot be reproduced by re-running their config, because they were never reproducible to begin with.
The last source of run-to-run variation. add_mz_noise_uniform/normal drew from rand::thread_rng, so which thread built a frame decided its jitter. Not cosmetic: a shifted m/z occasionally crosses a TOF bin boundary, which changes the (scan, tof) dedup and so the peaks written to analysis.tdf_bin. Two runs of an identical config differed by ~0.002 % of peaks even with the ground truth already reproducible. mscore gains `_with_rng` variants of both, and `add_mz_noise` is now generic over the RNG rather than pinned to ThreadRng. mscore::simulation::noise_rng derives a stream from a master seed plus the stable identifiers already in scope, so the draw does not depend on thread count or scheduling. The four sim builders carry a `noise_seed` and create one RNG per frame, keyed by (seed, frame_id): within a frame the spectra are built in a deterministic order, and frames are independent. 24 call sites across dia, dda, precursor and lazy_builder. `noise_seed` defaults to 0, which keeps the old thread_rng behaviour for API users who never set it; TimSim sets it from the run's sample_seed via create_frame_builder. Two runs of an identical config now produce a byte-identical analysis.tdf_bin — same sha256 over 93 MB — with the Frames table and every ground-truth table identical too. Tests: 38 mscore (5 for the stream derivation, 4 for the seeded jitter including that the jitter stays inside the requested ppm window and that different frames do not share a stream), 107 rustdf, 195 Python.
…c a worker The 4-thread run of the timing grid crashed on a NaN comparison in mscore's frame-occurrence search. It had passed in three earlier grid passes; seeding the global NumPy RNG changed which values are drawn and one draw landed in a degenerate region that was always reachable. Root cause: `np.random.beta` returns exactly 0.0 often enough to matter (22 of 50 000 draws at alpha=0.01), and `k_lower_rt` defaults to 0, so a zero sigma or lambda was reachable. `estimate_mu_from_mode_emg` divides by their product, so the erfcx target became inf, which passed the `y > 0` assert, and the Newton iteration returned NaN. That NaN flowed into the retention-time bounds, and `calculate_frame_occurrence_emg` unwrapped a `None` from `partial_cmp` on NaN. Three layers, root first: - Sampling: sigma and the k/lambda partner are clamped to a floor of 1e-12. Neither is a physical peak at zero and the downstream maths requires positive values. Note even the smallest non-zero draw observed was 1e-321, subnormal, whose reciprocal is inf anyway. - `erfcxinv`: the assert now requires finite as well as positive, so a degenerate parameter fails loudly rather than silently returning NaN. The Newton step is guarded where the derivative underflows: d/dx erfcx(x) tends to 0 as x grows, which made the unguarded iteration NaN for y below ~1.1e-8. The start value is already the asymptote there, so keeping the current iterate is both safe and accurate — verified identical to machine precision across y in [1e-8, 1e2] wherever the old code produced a finite answer. - mscore: non-finite bounds return an empty frame list, since a peptide with no finite elution window occupies no frames, and the closest-frame searches use `total_cmp` so they cannot panic whatever reaches them. The same `partial_cmp().unwrap()` hazard in the ion-mobility search is fixed too. Being deterministic is what surfaced this: before seeding, it could strike any run at low probability and look like bad luck. Tests: 208 Python passed / 15 skipped, 145 Rust passed. The new Python test is not vacuous — its beta parameters produce exact zeros, so it fails without the clamp.
Profiling the thread sweep showed FASTA processing flat at ~47 s from 8 to 120 threads, 26 % of total wall time at 120 threads and the ceiling on any further scaling. Inside it, `sample_peptides` was 33 s and identical at 50k and 250k peptides, which pointed at work proportional to something other than the requested complexity. The cause was `calculate_peptide_mono_isotopic_mass` rebuilding the 2 073-entry UniMod table, rebuilding the amino acid table and recompiling the UniMod regex on every call. One mass calculation cost ~190 us; 250 000 peptides cost 48 s. The same per-call `Regex::new` appeared in five more places across chemistry/utility.rs and data/peptide.rs. All of them are now `LazyLock` statics, built once. That alone takes 250 000 peptides from 47.7 s to 5.1 s (9.4x) and benefits every caller with no API change. The deliberate sort that keeps the sum bitwise reproducible is untouched. On top of that the connector gains `py_peptide.mono_isotopic_masses`, a batched rayon entry point, and `simulate_peptides` uses it: 0.51 s at 16 threads, 93x against the original, with masses bitwise identical to the per-peptide path and invariant across thread counts. It falls back to the per-peptide loop on an older connector. Note the submodule is an attribute of the extension module rather than an importable path, so `from imspy_connector.py_peptide import ...` raises ModuleNotFoundError; getting it off the module object is what makes the fast path actually reachable. The first version of this commit had the broken form and silently stayed on the slow path. End-to-end smoke, 16 threads: the FASTA stage falls 79.9 -> 43.3 s, of which `sample_peptides` is 76.8 -> 40.2 s. Remaining in that stage, and now dominant: neural retention-time prediction runs over the whole digested set to filter early eluters, before the run down-samples to the requested complexity. Every grid config digests 250 000 peptides whatever the target, so a 5k run predicts 250 000 retention times and discards 98 % of them. Reordering those two steps changes which peptides survive the filter, so it is a modelling decision rather than a mechanical fix, and is not done here. Tests: 214 Python passed / 15 skipped, 147 Rust passed, including that batched and per-peptide masses agree exactly and that modifications are still counted.
The digest produces `num_peptides_total` peptides regardless of the requested complexity — every grid config digests 250 000 — and the neural retention-time predictor ran over all of them so that the run could discard most of the output immediately afterwards. A 5k run predicted 250 000 retention times to keep 5 000; the smoke config predicted 250 000 to keep 2 000. `simulate_peptides` now takes the target count and thins the table before the predictor rather than after it. The thinning carries a 1.3x margin because the retention-time filter below removes part of the early-eluting population, and without headroom the table could fall under what the caller asked for. That filter is density-based — it thins the early bins down to the median bin count — so it behaves the same on a uniformly thinned table as on the full one. Runs at or above the digest target are untouched: the thinning only applies when the table is larger than the request, so the 250k and 500k datasets take exactly the path they did before. End-to-end smoke, 16 threads, 2 000 peptides kept: the FASTA stage falls 43.3 -> 6.5 s, its sampling lap 40.2 -> 3.4 s, and total wall 72 -> 35 s. This does change which peptides a small run selects, since the thinning now happens before the retention-time filter rather than after it. Reproducibility is unaffected: two runs of the same config still produce a byte-identical analysis.tdf_bin and identical ground-truth tables. Also fixes the accompanying log line, which subtracted the filtered length from itself and so always reported "Excluded 0 peptides with low retention times". Tests: 215 Python passed / 15 skipped.
…st tests The ordering test read a fasta from a developer machine and had no skip guard, so it failed in CI with FileNotFoundError. Ordering is a property of the digest, not of any particular proteome: it now runs on four synthetic proteins defined in the module. The parity test keeps the local HeLa subset when it is present and falls back to the same synthetic proteins otherwise, so CI exercises the batched digest instead of skipping it.
theGreatHerrLebert
added a commit
that referenced
this pull request
Sep 23, 2026
…ed CI (#487) * test(imspy-predictors): make the GMM fit test deterministic test_fit started its means at unseeded N(0, 1) draws. In about 4% of runs (8 of 200 measured) all three components collapsed into one cluster and 'assert 1 > 1' failed, which is what stopped the Python 3.12 job on main after #486 and skipped the imspy-simulation suite. Start the means at data points under a fixed seed: 0 of 200 runs collapse. * feat(imspy-simulation): depend on mzprov for output signing emit_provenance is on by default, but mzprov was not installable from PyPI, so a normal install silently skipped signing with a warning. mzprov 0.1.1 is now on PyPI: declare it, drop the ImportError fallbacks, and keep signing non-fatal. The .raw docstring's claim that mzprov cannot sign vendor .raw is out of date and goes. test_mzprov_signing runs each vendor path's signer (.d and mzML, sidecar and embedded; .raw) against mzprov's fixtures and checks the result with mzprov's verifier. * docs(imspy-simulation): mzprov 0.1.2's default key location mzprov 0.1.2 keeps new keys under ~/.config/mzprov/keys/ and uses an existing ~/.config/timsim/keys/ key in place, so TimSim users keep their signing identity. Require 0.1.2 and describe the default accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes the serial hot spots in the simulation pipeline, and fixes a
reproducibility defect found while verifying the speedups.
Releases
imspy-connector0.4.4,imspy-core0.4.4,imspy-simulation0.4.5.The Bruker SDK was forcing everything onto one thread
tims_index_to_mzis not safe to call concurrently on one handle. rustdf handledthat by serialising: whenever
use_bruker_sdkwas set, which is the default inevery config,
get_pasef_fragmentsandget_preprocessed_pasef_fragmentsignorednum_threadsentirely, and the DIA noise overlay and TDF writer ran frame by frame.Symptom that surfaced it: a seeded parallel overlay gave m/z differing by up to
4e-5 Th on ~8 % of peaks, in 3/8 runs at 4 threads and 7/8 at 32.
Parallel regions now read through
BrukerFormulaConverter, which evaluates thecalibration curves the
.dfile itself carries and isSync.It is only substituted where it is provably exact.
sdk_free_converter()isgated on
is_sdk_exact(): every frame must resolve to a real calibration row — nosilent fallback for NULL
T1/T2or an unsupported row — and everyMzCalibrationrow must be ModelType 1, the only model this converter reproduces bit-for-bit.
Mobility is exact under both; only the m/z axis is affected. ModelType 2 files stay
on the serial SDK path. Note this is per calibration, not per instrument or software
version — the same instrument on the same timsControl version produced both.
Batched TDF writer
Per-step profile over 700 real frames: SDK conversion 56 %, interleave 31 %, zstd
5 % — the phase named "convert+compress+write" was never compression-bound.
TDFWriter.write_framesnow hands a batch to rustdf, which converts, dedups(scan, tof), interleaves and compresses over a rayon pool. Only the append and therunning byte offset stay sequential.
Passing numpy arrays straight through is what makes it pay: the pre-existing
get_data_for_compression_partook Python lists, and the.tolist()marshallingalone was 21 % of writer time.
Output is byte-identical to the per-frame path (700/700 frames at 1, 4 and 16
threads). Writing the same frames both ways and comparing whole
.dfolders: theFramestable matches in all 18 columns including theTimsIdoffsets, and readingboth back gives 600/600 identical frames over 2.55 M peaks through the Bruker SDK
reader and the SDK-free reader. The files are equivalent, not bit-identical: 4 bytes
of 8.6 MB differ inside the zstd stream, decompressing to identical payloads,
because the Python
zstdmodule and the Rustzstdcrate link different libzstdbuilds.
Also fixed here:
zstd::encode_allomits the decompressed size from the frameheader, and the simple API every existing
.dwas written with then refuses to readthe frame. Uses
zstd::bulk::compress.Batched sage digest
generate_indexed_database()sets up sage's parallel index build, and it was calledonce per protein — ~3 000 core-seconds on a 120-core machine for a stage that ran
faster with 4 threads. Now one database per 2 000-protein chunk. Peptide sets per
protein are unchanged: sage attaches every protein a peptide occurs in.
A run was not reproducible from its config
Two runs of an identical config with the same
sample_seedshared 11 of ~1690peptides. Two independent causes:
sample_seedreached only the pandasdown-sample and the noise overlay. Everything deciding what gets simulated used
bare
np.random.*— peptide selection and abundance, EMG retention-time shape,ion-mobility and scan variance, phosphosite choice, the decoy draw — 16 call sites
in
timsim/jobs/alone.set, and the sampler draws from it withnp.random.choice. Set iteration order follows Python's per-process string hashseed, so the draw varied even once the RNG was seeded. Both now return sorted lists.
Every intermediate table is now bit-identical between runs — proteins, peptides,
ions, fragment_ions, frames, scans — and stays identical across thread counts (4 vs
16). Datasets generated before this cannot be reproduced by re-running their config;
they never could.
The m/z jitter is seeded too.
add_mz_noise_uniform/add_mz_noise_normaldrew fromrand::thread_rng, so which thread built a frame decided its jitter, and a shifted m/zoccasionally crossed a TOF bin boundary and changed the dedup. mscore gains
_with_rngvariants,mscore::simulation::noise_rngderives a stream from a master seed plus the identifiers already inscope, and the four sim builders carry a
noise_seedand use one RNG per frame keyed by(seed, frame_id). It defaults to 0, keeping the old behaviour for API users who never set it.Two runs of an identical config now produce a byte-identical
analysis.tdf_bin— same sha256over 93 MB — with the
Framestable and every ground-truth table identical, and the ground truthstays identical across thread counts (4 vs 16).
Not covered: reproducibility across different hardware. Retention time and fragment intensity come
from GPU models, so a different GPU or toggling
use_gpumay change the ground truth. Untested.Measured
Intel Xeon w9-3595X, 120 logical cores, RTX 5090, 60 min dia-PASEF, HeLa digest,
reading the reference from disk. Full grid of 25 runs per version.
Warm-up at 120 threads, 5k peptides: total 371 -> 90 s, frame assembly 238 -> 27 s,
of which the noise overlay is 87.4 -> 8.6 s and the writer 45.5 -> 13.8 s. The FASTA
stage went from ~3 000 to ~148 core-seconds. Replicates at 50k: CV 0.23 %.
Cost: peak RSS is up, 15 % at 250k and 23 % at 500k, because the parallel overlay
holds a batch of frames in flight. Capped runs still pass (250k completes under a
32 GB cap) but 500k now peaks at 64.8 GB uncapped.
Tests
195 Python passed / 15 skipped, 145 Rust passed. New coverage: batched-vs-per-protein
digest parity with and without decoys, digest output ordering, overlay determinism at
1/4/32 threads with the SDK enabled, SDK-vs-formula converter parity, writer
byte-identity at 1/4/16 threads, whole-
.d-folder equivalence, the ragged-input andmismatched-length error paths, the noise-stream derivation, and that the seeded jitter is
reproducible and stays inside the requested ppm window.