Skip to content

Add residual design-effect correction and aggregation calibration parameters - #146

Merged
ammarcsj merged 7 commits into
mainfrom
add_residual_deff_correction
Jul 31, 2026
Merged

Add residual design-effect correction and aggregation calibration parameters#146
ammarcsj merged 7 commits into
mainfrom
add_residual_deff_correction

Conversation

@ammarcsj

@ammarcsj ammarcsj commented Jul 31, 2026

Copy link
Copy Markdown
Member

Corrects the between-peptide correlation that residual-decorrelation pruning cannot remove, by measuring the excess sibling correlation over the cross-parent shuffle null and applying it as a Stouffer design effect (deff=1+(n-1)*rho).

It is additionally gated on that level's pre-pruning distance exceeding the tolerance, so when peptides are no more correlated than the shuffle null the gate stays closed, rho remains 0.0 and aggregation is bit-identical to before. A single level-wide excess rho is applied per level, clipped at zero once on the level mean rather than per parent, since per-parent means are noisy and clipping each would rectify that noise into a positive bias. Below a total-sample threshold the source switches to raw pre-pruning correlations, because at low replicate counts the survivor rho reads ~0 while the correlation still leaks into the Stouffer sum.

Also adds the calibration parameters this work required, each as its own commit:

  • median_normalization — take the median of the between-condition fold-change distribution as the shift instead of choosing between its median and its mode.
  • max_peptides_per_protein — replaces the hard-coded 31-peptide cap with a configurable one.
  • residual_decorrelation_cutoff_grid — the default grid now extends to -1.0; the negative part is reached only when no cutoff meets the tolerance, in which case the tightest prunes down to min_keep.
  • residual_decorr_corr_mode / _cap — bounds how many samples the sibling-correlation estimate may use, so pruning aggressiveness does not grow with sample count.
  • median_on_collapse — when pruning collapses a parent to a single surviving child, aggregate via the median of all eligible children rather than reporting the lone survivor. Since median_z does not scale by sqrt(n), no design effect applies on this path, and single-peptide parents are unaffected at any rho.
  • per-level diagnostic plots (before/after/null CDFs and the cutoff sweep trace) written when runtime_plots is set.

Behavior changes at default settings

  • max_peptides_per_protein defaults to no cap; the previous behavior was effectively a cap of 31.
  • The default cutoff grid extends further than before, so pruning can drop more siblings.
  • median_on_collapse and residual_deff_correction default to on.

Testing

tests/unit_tests: 185 passed, 1 skipped. Each commit parses on its own, so the history is bisectable.

@ammarcsj
ammarcsj merged commit 9538b08 into main Jul 31, 2026
4 checks passed
@ammarcsj
ammarcsj deleted the add_residual_deff_correction branch July 31, 2026 12:43
Comment on lines 69 to +71
df_c1_normed, df_c2_normed = aqnorm.normalize_if_specified(df_c1 = df_c1, df_c2 = df_c2, c1_samples = c1_samples, c2_samples = c2_samples, normalize_within_conds = runconfig.normalize, normalize_between_conds = runconfig.normalize,
runtime_plots = runconfig.runtime_plots, protein_subset_for_normalization_file=runconfig.protein_subset_for_normalization_file, pep2prot = pep2prot)#, "./test_data/normed_intensities.tsv")
runtime_plots = runconfig.runtime_plots, protein_subset_for_normalization_file=runconfig.protein_subset_for_normalization_file, pep2prot = pep2prot,
median_normalization = getattr(runconfig, 'median_normalization', False))#, "./test_data/normed_intensities.tsv")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please simplify this by splitting into several statements?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nevermind, now I got it..
ping me once it is a good time to format the codebase ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants