feat: preserve I/L in loader outputs

docs/api/datasets.md

@@ -87,11 +87,17 @@ dataset.save(Path("output_directory"))
 - **Filtering**: Removes invalid tokens and unsupported modifications
 - **Evaluation**: Computes correctness labels when ground truth available
 
+#### Leucine and isoleucine
+
+Winnow treats leucine and isoleucine as interchangeable when computing training labels (`correct`, `num_matches`) and beam edit distance because they are structural isomers, so standard mass spectrometry cannot differentiate them by precursor mass alone.
+
+By default, stored `prediction`, labelled `sequence`, and beam `ScoredSequence.sequence` keep the source residue (`L` or `I`). Set `output_leucine_as_isoleucine=True` on any `DatasetLoader` (or `data_loader.output_leucine_as_isoleucine=true` in the CLI) to restore the legacy behaviour of rewriting stored tokens with `L` → `I`.
+
 ### Data loaders
 
 The datasets module provides several data loaders that implement the `DatasetLoader` protocol.
 
-All loaders require `residue_masses` (ProForma residue → mass; see `configs/residues.yaml`). Loaders that ingest tool-specific peptide notation also take `residue_remapping`.
+All loaders require `residue_masses` (ProForma residue → mass; see `configs/residues.yaml`). Loaders that ingest tool-specific peptide notation also take `residue_remapping`. Each loader accepts `output_leucine_as_isoleucine` (default `false`; see above).
 
 #### InstaNovoDatasetLoader
 

docs/api/features/beam.md

@@ -56,6 +56,6 @@ The dataset must have beam predictions available (`dataset.predictions` must not
 - When beam size is 1, margin and entropy default to 0; `edit_distance` defaults to `1.0`
 - Entropy is computed on the **normalised** runner-up probabilities (excluding top-1)
 - Z-score uses the full beam including top-1 for mean/std calculation
-- Edit distance collapses I/L via the same leucine normalisation used elsewhere in the package (`L` → `I`)
+- Edit distance collapses I/L at comparison time (`L` → `I` on copies of the top-1 and top-2 token lists); stored beam sequences are not rewritten unless `data_loader.output_leucine_as_isoleucine=true`
 - Both sequences empty is treated as undefined (`1.0`), not as raw Levenshtein zero
 - All probability values are derived from `exp(sequence_log_probability)`

docs/cli.md

@@ -215,7 +215,7 @@ Set `diagnostics.fail_on_warning=true` to exit with code 1 when $|\widehat{\math
 
 ### `winnow annotate-proteome-hits`
 
-Load spectra and *de novo* predictions via a `DatasetLoader`, drop peptides shorter than a residue threshold, and add a boolean `proteome_hit` column by matching each prediction (modifications stripped, isoleucine normalised to leucine) as a substring of a reference proteome. The annotated holdout is written as a Winnow dataset directory (`metadata.csv` plus optional `predictions.pkl`).
+Load spectra and *de novo* predictions via a `DatasetLoader`, drop peptides shorter than a residue threshold, and add a boolean `proteome_hit` column by substring matching against a reference proteome. Matching collapses leucine and isoleucine to the same residue on both the prediction (after stripping modifications) and each FASTA sequence. Stored peptide sequences in the output keep their original residue identity unless you set `data_loader.output_leucine_as_isoleucine=true`. The annotated holdout is written as a Winnow dataset directory (`metadata.csv` plus optional `predictions.pkl`).
 
 This produces the `proteome_hit` column consumed by `diagnose-calibration` with `data_loader=winnow` and `diagnostics.label_source=precomputed diagnostics.label_column=proteome_hit`. Training does not use `proteome_hit`; the column may sit alongside sequence-derived labels without changing train behaviour. Prediction accepts datasets with or without the column.
 

docs/configuration.md

@@ -532,7 +532,7 @@ You must choose exactly one labelling mode. The command validates config before
 
 ### Main config (`configs/annotate_proteome_hits.yaml`)
 
-Loads a *de novo* prediction dataset, filters short peptides, adds `proteome_hit` via FASTA substring matching, and writes a Winnow dataset directory. See the [CLI reference](cli.md#winnow-annotate-proteome-hits).
+Loads a *de novo* prediction dataset, filters short peptides, adds `proteome_hit` via FASTA substring matching, and writes a Winnow dataset directory, stripping post-translational modifications and treating leucine and isoleucine as equivalent. See the [CLI reference](cli.md#winnow-annotate-proteome-hits).
 
 ```yaml
 defaults:
@@ -594,6 +594,7 @@ Each data format has a dedicated loader configuration in `configs/data_loader/`:
 _target_: winnow.datasets.data_loaders.InstaNovoDatasetLoader
 add_index_cols: false  # If true, add experiment_name + spectrum_id for parquet/ipc (InstaNovo-style). MGF always gets these columns.
 residue_masses: ${residue_masses}
+output_leucine_as_isoleucine: false  # false: L/I as in source in exports; true: L shown as I in output tokens (legacy)
 residue_remapping:
   "M(ox)": "M[UNIMOD:35]"
   "C(+57.02)": "C[UNIMOD:4]"

tests/datasets/data_loaders/test_instanovo.py

@@ -17,6 +17,7 @@ def _finalize(loader, metadata, *, has_labels: bool = True):
         loader.metrics,
         has_labels=has_labels,
         residue_remapping=loader.metrics.residue_set.residue_remapping,
+        output_leucine_as_isoleucine=loader.output_leucine_as_isoleucine,
     )
 
 
@@ -542,8 +543,33 @@ def test_process_beams_produces_scored_sequences(self, loader):
         assert hasattr(beams[0][0], "sequence")
         assert hasattr(beams[0][0], "sequence_log_probability")
 
-    def test_process_beams_replaces_l_with_i_in_sequences(self, loader):
-        """L amino acid must be replaced with I at token level after split."""
+    def test_process_beams_preserves_l_by_default(self, loader):
+        beam_df = pl.DataFrame(
+            {
+                "predictions_beam_0": ["PEPTLDE"],
+                "predictions_log_probability_beam_0": [-0.5],
+                "predictions_token_log_probabilities_0": [
+                    "[-0.1, -0.2, -0.3, -0.4, -0.5, -0.6, -0.7]"
+                ],
+            }
+        )
+        beams = loader._process_beams(beam_df)
+        assert beams[0] is not None
+        assert "L" in beams[0][0].sequence
+
+    def test_process_beams_replaces_l_with_i_when_legacy_flag(
+        self, full_residue_masses, standard_remapping
+    ):
+        loader = InstaNovoDatasetLoader(
+            residue_masses=full_residue_masses,
+            residue_remapping=standard_remapping,
+            output_leucine_as_isoleucine=True,
+            beam_columns={
+                "sequence": "predictions_beam_",
+                "log_probability": "predictions_log_probability_beam_",
+                "token_log_probabilities": "predictions_token_log_probabilities_",
+            },
+        )
         beam_df = pl.DataFrame(
             {
                 "predictions_beam_0": ["PEPTLDE"],
@@ -799,7 +825,27 @@ def test_process_predictions_splits_comma_separated_prediction(self, loader):
         result = loader._process_predictions(preds_df, ["spectrum_id"])
         assert result["prediction"].iloc[0] == ["P", "E", "P", "T", "I", "D", "E"]
 
-    def test_finalize_replaces_l_with_i_in_prediction_list(self, loader):
+    def test_finalize_preserves_l_in_prediction_list_by_default(self, loader):
+        preds_df = pd.DataFrame(
+            {
+                "spectrum_id": [1],
+                "predictions": ["PEPTLDE"],
+                "predictions_tokenised": ["P, E, P, T, L, D, E"],
+                "log_probs": [-0.5],
+            }
+        )
+        processed = loader._process_predictions(preds_df, ["spectrum_id"])
+        finalized = _finalize(loader, processed, has_labels=False)
+        assert "L" in finalized["prediction"].iloc[0]
+
+    def test_finalize_replaces_l_with_i_when_legacy_flag(
+        self, full_residue_masses, standard_remapping
+    ):
+        loader = InstaNovoDatasetLoader(
+            residue_masses=full_residue_masses,
+            residue_remapping=standard_remapping,
+            output_leucine_as_isoleucine=True,
+        )
         preds_df = pd.DataFrame(
             {
                 "spectrum_id": [1],

tests/datasets/data_loaders/test_mztab.py

@@ -18,15 +18,21 @@ def _finalize(loader, metadata, *, has_labels: bool = True):
         loader.metrics,
         has_labels=has_labels,
         residue_remapping=loader.metrics.residue_set.residue_remapping,
+        output_leucine_as_isoleucine=loader.output_leucine_as_isoleucine,
     )
 
 
-def _normalize_peptide(loader, value: object) -> list[str] | None:
+def _normalize_peptide(
+    loader, value: object, *, output_leucine_as_isoleucine: bool | None = None
+) -> list[str] | None:
     """Normalize a peptide cell using loader metrics (string or token list)."""
+    if output_leucine_as_isoleucine is None:
+        output_leucine_as_isoleucine = loader.output_leucine_as_isoleucine
     return data_utils.normalize_peptide_cell(
         value,
         loader.metrics,
         residue_remapping=loader.metrics.residue_set.residue_remapping,
+        output_leucine_as_isoleucine=output_leucine_as_isoleucine,
     )
 
 
@@ -428,9 +434,16 @@ def test_process_predictions_preserves_modification_names_in_untokenised(
         )
         assert result["prediction_untokenised"][0] == "PEPTC[Carbamidomethyl]DE"
 
-    def test_normalize_replaces_leucine_at_token_level(self, loader):
+    def test_normalize_preserves_leucine_by_default(self, loader):
         tokens = _normalize_peptide(loader, "PEPTLDE")
         assert tokens is not None
+        assert "L" in tokens
+
+    def test_normalize_replaces_leucine_when_legacy_flag(self, loader):
+        tokens = _normalize_peptide(
+            loader, "PEPTLDE", output_leucine_as_isoleucine=True
+        )
+        assert tokens is not None
         assert "L" not in tokens
         assert "I" in tokens
 
@@ -504,8 +517,7 @@ def test_normalize_remaps_carbamidomethyl_without_corrupting_name(self, loader):
         tokens = _normalize_peptide(loader, "GEEHC[Carbamidomethyl]GHLLQAHK")
         assert tokens is not None
         assert "C[UNIMOD:4]" in tokens
-        assert "L" not in tokens
-        assert tokens.count("I") == 2
+        assert tokens.count("L") == 2
 
     def test_normalize_unmodified_sequence_passes_through(self, loader):
         assert _normalize_peptide(loader, "ACGM") == ["A", "C", "G", "M"]
@@ -893,6 +905,16 @@ def test_evaluate_correct_flag_false_on_different_sequence(self, loader):
         result = _finalize(loader, metadata, has_labels=True)
         assert not result["correct"][0]
 
+    def test_evaluate_correct_flag_true_on_l_vs_i_match(self, loader):
+        metadata = pl.DataFrame(
+            {
+                "sequence": [["P", "E", "P", "T", "L", "D", "E"]],
+                "prediction": [["P", "E", "P", "T", "I", "D", "E"]],
+            }
+        )
+        result = _finalize(loader, metadata, has_labels=True)
+        assert result["correct"][0]
+
     def test_evaluate_correct_flag_false_on_length_mismatch(self, loader):
         metadata = pl.DataFrame(
             {"sequence": [["P", "E", "P"]], "prediction": [["P", "E"]]}

tests/datasets/data_loaders/test_primenovo.py

@@ -376,7 +376,24 @@ def test_load_tokenises_and_remaps_via_finalize(self, loader, mgf_path, tsv_path
         assert "M[UNIMOD:35]" in row["prediction"]
         assert "M[+15.995]" in row["prediction_untokenised"]
 
-    def test_load_applies_token_level_l_to_i(self, loader, mgf_path, tsv_path):
+    def test_load_preserves_l_in_prediction_by_default(
+        self, loader, mgf_path, tsv_path
+    ):
+        dataset = loader.load(data_path=mgf_path, predictions_path=tsv_path)
+        meta = dataset.metadata
+        row = meta.set_index("spectrum_id").loc["spectra:run_X_SCANS_1"]
+        assert "L" in row["prediction"]
+        assert "L" in row["prediction_untokenised"]
+
+    def test_load_applies_token_level_l_to_i_when_legacy_flag(
+        self, full_residue_masses, mgf_path, tsv_path
+    ):
+        loader = PrimeNovoDatasetLoader(
+            residue_masses=full_residue_masses,
+            residue_remapping=_PRIMENOVO_REMAPPING,
+            mid_sequence_n_terminal_mods=_MID_SEQUENCE_N_TERMINAL_MODS,
+            output_leucine_as_isoleucine=True,
+        )
         dataset = loader.load(data_path=mgf_path, predictions_path=tsv_path)
         meta = dataset.metadata
         row = meta.set_index("spectrum_id").loc["spectra:run_X_SCANS_1"]

tests/datasets/data_loaders/test_utils.py

@@ -16,6 +16,7 @@
     "P": 97.052764,
     "E": 129.042593,
     "L": 113.084064,
+    "I": 113.084064,
 }
 REMAPPING = {"M[Oxidation]": "M[UniMod:35]"}
 
@@ -107,15 +108,24 @@ def test_string_splits_and_remaps(self, metrics: Metrics) -> None:
         )
         assert tokens == ["A", "G"]
 
-    def test_leucine_mapped_at_token_level(self, metrics: Metrics) -> None:
+    def test_leucine_preserved_by_default(self, metrics: Metrics) -> None:
         tokens = data_utils.normalize_peptide_cell(
             ["A", "L"],
             metrics,
             residue_remapping=REMAPPING,
         )
+        assert tokens == ["A", "L"]
+
+    def test_leucine_mapped_when_legacy_flag_true(self, metrics: Metrics) -> None:
+        tokens = data_utils.normalize_peptide_cell(
+            ["A", "L"],
+            metrics,
+            residue_remapping=REMAPPING,
+            output_leucine_as_isoleucine=True,
+        )
         assert tokens == ["A", "I"]
 
-    def test_string_leucine_mapped_without_corrupting_mod_names(
+    def test_string_leucine_legacy_without_corrupting_mod_names(
         self, metrics: Metrics
     ) -> None:
         """Uppercase L inside a modification name must not be rewritten before split."""
@@ -124,6 +134,7 @@ def test_string_leucine_mapped_without_corrupting_mod_names(
             "AC[Labelled]L",
             metrics,
             residue_remapping=remapping,
+            output_leucine_as_isoleucine=True,
         )
         assert tokens == ["A", "C[UniMod:999]", "I"]
 
@@ -294,6 +305,26 @@ def test_row_num_matches_and_correct(self, metrics: Metrics) -> None:
             1, ["A", "G"], ["A", "G"], sequence_valid=True, prediction_valid=True
         )
 
+    def test_row_num_matches_treats_l_and_i_as_equivalent(
+        self, metrics: Metrics
+    ) -> None:
+        sequence = ["A", "L", "G"]
+        prediction = ["A", "I", "G"]
+        assert data_utils.row_num_matches(
+            sequence,
+            prediction,
+            metrics,
+            sequence_valid=True,
+            prediction_valid=True,
+        ) == len(sequence)
+        assert data_utils.row_is_correct(
+            len(sequence),
+            sequence,
+            prediction,
+            sequence_valid=True,
+            prediction_valid=True,
+        )
+
 
 class TestFinalizePeptideMetadata:
     @pytest.fixture()

tests/utils/test_proteome.py

@@ -15,6 +15,7 @@
 from winnow.utils.config_path import get_config_dir
 from winnow.utils.proteome import (
     annotate_calibration_dataset,
+    normalize_sequence,
     processed_peptide_for_match,
     residue_set_from_residues_yaml,
     residue_token_count,
@@ -33,16 +34,16 @@ def residue_masses(residue_set: ResidueSet) -> dict[str, float]:
 
 def test_processed_peptide_for_match_strips_mods() -> None:
     cases = [
-        ("PEP(+123.45)TIDE[UNIMOD:35]K", "PEPTLDEK"),
-        ("[UNIMOD:1]-PEP(+123.45)TIDE[UNIMOD:35]K", "PEPTLDEK"),
+        ("PEP(+123.45)TIDE[UNIMOD:35]K", "PEPTIDEK"),
+        ("[UNIMOD:1]-PEP(+123.45)TIDE[UNIMOD:35]K", "PEPTIDEK"),
         ("PEP(foo)TAGDE", "PEPTAGDE"),
         ("(+47.01)-PEPTAGDE", "PEPTAGDE"),
         ("PEPTAGDE[Carboxyl]", "PEPTAGDE"),
         ("[Acetyl]-PEPTAGDE", "PEPTAGDE"),
         ("(N-term)PEPTAGDE", "PEPTAGDE"),
-        ("PEP+15.99TIDE", "PEPTLDE"),
-        ("PEP-15.99TIDE", "PEPTLDE"),
-        ("PEP15.99TIDE", "PEPTLDE"),
+        ("PEP+15.99TIDE", "PEPTIDE"),
+        ("PEP-15.99TIDE", "PEPTIDE"),
+        ("PEP15.99TIDE", "PEPTIDE"),
         ("+42.01-PEPTAGDE", "PEPTAGDE"),
     ]
     for raw, expected in cases:
@@ -52,6 +53,44 @@ def test_processed_peptide_for_match_strips_mods() -> None:
         assert out == expected, raw
 
 
+def test_processed_peptide_for_match_maps_leucine_to_isoleucine() -> None:
+    assert processed_peptide_for_match("PEPTLDE") == "PEPTIDE"
+
+
+def test_processed_peptide_for_match_unimod_stripped_before_leucine_map() -> None:
+    """Bracketed UNIMOD must be removed before L → I so names are not corrupted."""
+    assert processed_peptide_for_match("PEPTC[UNIMOD:4]DE") == "PEPTCDE"
+
+
+def test_normalize_sequence_maps_leucine_to_isoleucine() -> None:
+    assert normalize_sequence("PEPTLDE") == "PEPTIDE"
+    assert normalize_sequence("PEPTIDE") == "PEPTIDE"
+
+
+def test_annotate_l_query_hits_fasta_with_isoleucine(
+    residue_set: ResidueSet, tmp_path: Path
+) -> None:
+    dataset = CalibrationDataset(
+        metadata=pd.DataFrame(
+            {
+                "spectrum_id": ["a"],
+                "prediction": [["P", "E", "P", "T", "L", "D", "E"]],
+                "prediction_untokenised": ["PEPTLDE"],
+                "confidence": [0.9],
+                "mz_array": [[100.0]],
+                "intensity_array": [[1.0]],
+            }
+        )
+    )
+    fasta_path = tmp_path / "proteome.fasta"
+    fasta_path.write_text(">prot1\nXXXXXXMKLLPEPTIDEMKLLYYYY\n")
+
+    annotated, _stats = annotate_calibration_dataset(
+        dataset, fasta_path, residue_set, min_residue_length=7
+    )
+    assert annotated.metadata["proteome_hit"].tolist() == [True]
+
+
 @pytest.mark.parametrize(
     ("prediction", "expected"),
     [

winnow/calibration/features/beam.py

@@ -8,7 +8,7 @@
 from winnow.calibration.features.base import CalibrationFeatures, FeatureDependency
 from winnow.datasets.calibration_dataset import CalibrationDataset
 from winnow.calibration.features.utils import require_beam_predictions
-from winnow.datasets.data_loaders.utils import _normalize_leucine_tokens
+from winnow.datasets.data_loaders.utils import replace_leucine_with_isoleucine
 
 
 def _beam_len(beam: Optional[List]) -> int:
@@ -75,8 +75,8 @@ def _beam_edit_distance(beam: Optional[List]) -> float:
     if not top_seq and not second_seq:
         return 1.0
     return _normalised_levenshtein(
-        _normalize_leucine_tokens(top_seq),
-        _normalize_leucine_tokens(second_seq),
+        replace_leucine_with_isoleucine(top_seq),
+        replace_leucine_with_isoleucine(second_seq),
     )
 
 

winnow/configs/data_loader/instanovo.yaml

@@ -7,6 +7,9 @@ _target_: winnow.datasets.data_loaders.InstaNovoDatasetLoader
 add_index_cols: false
 
 residue_masses: ${residue_masses}
+# false: prediction, sequence, and beams keep L and I from the source.
+# true: stored tokens show I wherever the source had L (legacy).
+output_leucine_as_isoleucine: false
 residue_remapping:  # Used to map InstaNovo legacy notations to UNIMOD tokens.
   "M(ox)": "M[UNIMOD:35]"  # Oxidation
   "M(+15.99)": "M[UNIMOD:35]"  # Oxidation