Train a calibrator on InstaNovo v1.3.0 predictions: the general model is fitted on v1.2.0's #262

Why

InstaDeepAI/winnow-general-model was fitted on predictions from the released InstaNovo checkpoint (#57–#61), and its configuration fits that checkpoint: its unsupported_residues ([UNIMOD:5], [UNIMOD:385], (+25.98)) are the residues of instanovo-v1.2.0 that Prosit can't score (instadeepai/INFlow#158).

Six of its ten features come from the checkpoint's own scores: margin, median_margin, entropy, z-score, min_token_probability and std_token_probability. A different checkpoint produces differently distributed scores, so the calibrated probabilities, and the FDR derived from them, no longer mean what they mean for v1.2.0.

InstaNovo v1.3.0 is that different checkpoint: the 133-residue release candidate, AIchor experiment 33768d55-ac95-40ab-9545-b3889ef8facf, sha256 5443d0cc97d5e57d1b220570d9da78c568f3812b44ff5b85c881f4e3befb2937, residue set in InstaNovo-internal#743. In INFlow's comparison of the two checkpoints on three samples, v1.3.0 passes 3.0–9.2 percentage points fewer PSMs at 5% FDR, even on spectra both can be scored on. With a calibrator fitted on v1.2.0, that can't be attributed to the model rather than to miscalibration.

Asked for

A calibrator fitted on v1.3.0 predictions, published beside the general model and named after the checkpoint it pairs with.

Choices to make when fitting:

• Training data: the same corpus as the general model (feat: prepare training data for general model #58), predicted again with the v1.3.0 checkpoint, so the two calibrators differ only in the predictions.
• Residues: the checkpoint's 133, with masses from its own residue set. That depends on Mass error raises a bare KeyError on any residue the loaded calibrator has no mass for #254 (fix in fix: take residue masses for a wider vocabulary from the configured set #255) and koina.constraints.unsupported_residues has no effect when a pretrained calibrator is loaded #252 (fix in fix: let koina.constraints.unsupported_residues reach a loaded calibrator #253).
• Fragment and iRT model: Prosit represents 53 of the 133 residues, AlphaPeptDeep 130. With Prosit, 22–36% of v1.3.0's PSMs are excluded before calibration. Fitting on AlphaPeptDeep features (which needs koinapy cannot merge predictions from a model whose output width varies #256/fix: ask Koina for one batch at a time so ragged outputs merge #258 and koina.intensity_model and koina.irt_model have no effect on a loaded calibrator #257/fix: let a loaded calibrator be pointed at a different Koina model #259) would keep them, and would also give AlphaPeptDeep a calibrator fitted for it, which no Koina model other than Prosit has today.
• learn_from_missing: fitted with it on, the missingness indicators enter the network, and PSMs Koina can't score are imputed instead of dropped. The published model was fitted with it off, so turning it on at prediction time can't help it.
• Limits: max_peptide_length is 30 in the general model's features, while INFlow decodes up to 40.

Done when

• Calibration on held-out labelled data is reported: reliability curve and ECE, against the general model on v1.2.0.
• The FDR is checked against a database search or entrapment (instadeepai/INFlow#41).
• The calibrator is on Hugging Face, with the checkpoint it pairs with in its model card.

Used by instadeepai/INFlow#185.