Check the false-positive cost of trying many variants
How does testing twenty null variants change the chance of declaring at least one apparent improvement?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
What to compare
For independent uniform null p-values, compare against 1-(1-t)^20. FWER and FDR are different targets; do not substitute one for the other.
Useful outputs
- A frozen family of comparisons and an explicitly named error rate.
- Analytical independent-null FWER and descriptive simulated proportions.
- A separate extension plan for correlated comparisons or false-discovery-rate control.
Inputs and prerequisites
- Original worked jobs use the qualified standard-library path.
- Additional statistical packages and actual model evaluations need separately quoted capabilities.
What this work would not establish
- The independent uniform null model does not represent every adaptive model-selection process.
- The proposed extension is unrun.
Start with these sources.
Familywise error with independent null p-values
Compare unadjusted and Bonferroni thresholds on fixed-size families of synthetic uniform null p-values, with analytical familywise probabilities.
An exact small-sample permutation reference
Enumerate every balanced assignment for two groups of four observations and compare the exact two-sided randomization p-value with a sampled estimate.
false_discovery_control
A multiple-testing reference for adjusted p-values using Benjamini–Hochberg and Benjamini–Yekutieli procedures.
statsmodels
Statistical model estimation, hypothesis tests, diagnostics, and time-series analysis.
Make the question your own.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.