Repository navigation
Account for arm selection in switch-rate confidence intervals - #3
Open
infinityscroll wants to merge 1 commit into
Open
infinityscroll wants to merge 1 commit into
infinityscroll wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The switch score is the minimum observed arm rate, but its interval currently belongs only to whichever arm attains that minimum. With tied rates such as 5/10 and 50/100, reversing arm labels can change the reported interval without changing the data or estimand.
This adds simultaneous Clopper–Pearson bounds and takes the minimum lower and upper endpoints. Each required arm receives a two-sided error budget of 0.05/m. The construction gives at least 95% coverage under fixed-arm marginal binomial sampling; it does not require independence between arms, and it does not address within-arm clustering or post-hoc selection of families/models. The implementation stays standard-library-only.
The default rejects incomplete required-arm sets.
legacy-wilsonpreserves the historical behavior, andrepro/precompute_cells.pyexplicitly selects that mode. Point estimates and tallies remain unchanged for complete arm sets; shipped paper data and figure sources are untouched. CLI JSON records the interval method.Validation: all 85 tests pass, including the published reference-disagreement replay; Ruff and
git diff --checkpass. Added tests cover binomial boundaries, exact finite-sample enumeration, tied-arm order invariance, missing arms, legacy equivalence and CLI serialization. An independent local comparison against SciPy exact binomial intervals passed 34 boundary/central cases up to n=100,000, with maximum absolute endpoint difference 2.83e-14. This numerical comparison is not a machine-verified coverage proof.