An exact small-sample permutation reference
Enumerate every balanced assignment for two groups of four observations and compare the exact two-sided randomization p-value with a sampled estimate.
Proposed original example. Source release and any execution are separate steps.
Can a Monte Carlo permutation implementation be checked against a fully enumerated finite reference?
Evaluate all 70 assignments using Fraction arithmetic and an explicitly inclusive tie rule; record the exchangeability assumption separately.
What you could produce
- A scoped observations JSON record and a comparison with the stated reference.
Before you use it
- Python 3.13 standard library
Limits to keep in view
- No research code was executed by the preparation tool.
- Author output cannot issue an independent scientific-verification result.
Source and permission context
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Catalog listing reviewed. This review covers the description and source links displayed here.
Local draft title, collective byline, original summary, and proposed split terms reviewed.
Reviewed 2026-09-14. Copying or adapting source files remains subject to their own terms.
Before copying source material
- Proposed local-draft licenses: original code MIT, explanations CC-BY-4.0, synthetic numeric data CC0-1.0; publication/disclosure approval remains separate.
Put this resource to work.
Put uncertainty around a small agent-success benchmark
How misleading can a nominal 95% success-rate interval be when only twenty independent tasks are observed?
Check the false-positive cost of trying many variants
How does testing twenty null variants change the chance of declaring at least one apparent improvement?