Put uncertainty around a small agent-success benchmark
How misleading can a nominal 95% success-rate interval be when only twenty independent tasks are observed?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
Make the question your own.
5 sources · Exact preparation context
- Exact binomial coverage of Wald and Wilson intervals →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: c3675952145f7c18cb07be1d98419d0aac99b5238319a042b7f7fe157f88389b
- Conditional moments of a bootstrap mean generator →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: d6544b6b0742d8d51f543d0ee3e90aada638d21e406773fdba467ba1efdf307f
- An exact small-sample permutation reference →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: 41fc29009376f57de6e399dbdea5ac25eeef03a85e8ff3be88b0c6cc9362b0ad
- bootstrap →
SciPy documentation. bootstrap. https://docs.scipy.org/doc/scipy-1.18.0/reference/generated/scipy.stats.bootstrap.html; observed version 1.18.0.
Record revision: e4ffc1a5eb1aab3d36cf2c58d943cff750c9e55c060fcc721184913eab805c84
- permutation_test →
SciPy documentation. permutation_test. https://docs.scipy.org/doc/scipy-1.18.0/reference/generated/scipy.stats.permutation_test.html; observed version 1.18.0.
Record revision: e6681e6141a227a65d80da4fe8ebab552f5a4538d382c0c1924817437163c01b
Source links open the current listing. This plan retains the revisions above; listing review does not grant permission to copy or adapt source files.
Before carrying out this plan
- Original pilot jobs require the existing qualified isolated standard-library runtime.
- SciPy extensions require a separately qualified dependency environment.
Limits to keep in view
- This is a worked model of benchmark uncertainty, not an evaluation of a named AI model.
- No sampler-calibration verdict or completed scientific check is implied.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.
What to compare
Keep n=20,p=0.05 and all 21 success counts fixed for the worked case; compare coverage with the nominal level. Do not infer coverage for correlated benchmark tasks from this binomial model.
Useful outputs
- Exact count-enumeration table for the fixed binomial model, with both interval methods.
- A separate Monte Carlo table with its fixed seed, replicate count and uncertainty interpretation.
- A transfer note stating when repeated agents/tasks violate the independent-trials model.
Inputs and prerequisites
- Original pilot jobs require the existing qualified isolated standard-library runtime.
- SciPy extensions require a separately qualified dependency environment.
What this work would not establish
- This is a worked model of benchmark uncertainty, not an evaluation of a named AI model.
- No sampler-calibration verdict or completed scientific check is implied.
Start with these sources.
Exact binomial coverage of Wald and Wilson intervals
Executable Science local pilot drafts
Enumerate all binomial outcomes at n=20 and p=0.05, then compare exact interval-coverage calculations with a fixed-seed simulation.
Investigate Do nominal 95% intervals achieve that coverage in this prespecified small-sample setting?
Conditional moments of a bootstrap mean generator
Executable Science local pilot drafts
Resample a fixed twenty-value empirical distribution and compare bootstrap moments with their conditional analytical values.
Investigate How do observed bootstrap moments compare with their conditional expectation and variance?
An exact small-sample permutation reference
Executable Science local pilot drafts
Enumerate every balanced assignment for two groups of four observations and compare the exact two-sided randomization p-value with a sampled estimate.
Investigate Can a Monte Carlo permutation implementation be checked against a fully enumerated finite reference?
bootstrap
SciPy documentation
The bootstrap reference explains percentile, basic and bias-corrected accelerated confidence intervals, paired resampling and computation controls.
Investigate How do interval coverage and width compare for a known skewed distribution at small sample sizes?
permutation_test
SciPy documentation
A reference for permutation tests that distinguishes exchangeability structures, exact enumeration and randomized null sampling.
Investigate For a small paired dataset, does the randomized test approach an independently enumerated exact null distribution?