Check the false-positive cost of trying many variants
How does testing twenty null variants change the chance of declaring at least one apparent improvement?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
Make the question your own.
4 sources · Exact preparation context
- Familywise error with independent null p-values →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: c9a7be018fa87df84cd387ef16fd0b484ec9aae3d244a79e1dd417cf642c484d
- An exact small-sample permutation reference →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: 41fc29009376f57de6e399dbdea5ac25eeef03a85e8ff3be88b0c6cc9362b0ad
- false_discovery_control →
SciPy documentation. false_discovery_control. https://docs.scipy.org/doc/scipy-1.18.0/reference/generated/scipy.stats.false_discovery_control.html; observed version 1.18.0.
Record revision: 62a6dfc849fc86c5ea2d8df37a4219088705a1301c12a601377bf2ccdbb61d71
- statsmodels →
Preserve the upstream project name, version, repository link, applicable notices, and contributor attribution when preparing an artifact for reuse.
Record revision: 8631d38829d71396b4eb04f173ec9c006b033288e04430836060508714e13818
Source links open the current listing. This plan retains the revisions above; listing review does not grant permission to copy or adapt source files.
Before carrying out this plan
- Original worked jobs use the qualified standard-library path.
- Additional statistical packages and actual model evaluations need separately quoted capabilities.
Limits to keep in view
- The independent uniform null model does not represent every adaptive model-selection process.
- The proposed extension is unrun.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.
What to compare
For independent uniform null p-values, compare against 1-(1-t)^20. FWER and FDR are different targets; do not substitute one for the other.
Useful outputs
- A frozen family of comparisons and an explicitly named error rate.
- Analytical independent-null FWER and descriptive simulated proportions.
- A separate extension plan for correlated comparisons or false-discovery-rate control.
Inputs and prerequisites
- Original worked jobs use the qualified standard-library path.
- Additional statistical packages and actual model evaluations need separately quoted capabilities.
What this work would not establish
- The independent uniform null model does not represent every adaptive model-selection process.
- The proposed extension is unrun.
Start with these sources.
Familywise error with independent null p-values
Executable Science local pilot drafts
Compare unadjusted and Bonferroni thresholds on fixed-size families of synthetic uniform null p-values, with analytical familywise probabilities.
Investigate How does applying twenty tests change false-positive risk under a declared independent-null model?
An exact small-sample permutation reference
Executable Science local pilot drafts
Enumerate every balanced assignment for two groups of four observations and compare the exact two-sided randomization p-value with a sampled estimate.
Investigate Can a Monte Carlo permutation implementation be checked against a fully enumerated finite reference?
false_discovery_control
SciPy documentation
A multiple-testing reference for adjusted p-values using Benjamini–Hochberg and Benjamini–Yekutieli procedures.
Investigate How do the two procedures differ on fixed p-value vectors and simulated dependent null tests?
statsmodels
Upstream project repository
Statistical model estimation, hypothesis tests, diagnostics, and time-series analysis.
Investigate How do reported least-squares coefficients and standard errors behave as synthetic regressors become nearly collinear?