Statistics & uncertainty
Understand small samples and the assumptions behind an apparent improvement.
Choose a useful starting point.
A success rate from twenty tasks contains less information than the same rate from a large study. The binomial brief gives a fixed independent-trial model for comparing interval rules. Correlated tasks, repeated prompts and unequal task difficulty require a different model; an interval formula alone does not settle those choices.
Trying many variants also changes how an apparent improvement should be interpreted. Declare the family of comparisons and distinguish the chance of any false positive from the false-discovery rate. The resources below connect those questions to source methods and unexecuted plans.
Questions to work through.
Put uncertainty around a small agent-success benchmark
How misleading can a nominal 95% success-rate interval be when only twenty independent tasks are observed?
Check the false-positive cost of trying many variants
How does testing twenty null variants change the chance of declaring at least one apparent improvement?
Design a generative-model calibration check
Can inference recover declared generating quantities without conflating posterior predictive fit and parameter calibration?
Explore the sources.
All 195 resources →Iris
UCI Machine Learning Repository
Flower measurements for a compact, interpretable multiclass baseline and leakage audit.
Investigate How much do iris classification results change when preprocessing is restricted to training folds?
Exact binomial coverage of Wald and Wilson intervals
Executable Science local pilot drafts
Enumerate all binomial outcomes at n=20 and p=0.05, then compare exact interval-coverage calculations with a fixed-seed simulation.
Investigate Do nominal 95% intervals achieve that coverage in this prespecified small-sample setting?
SciPy
Upstream project repository
Numerical routines for integration, differential equations, optimization, transforms, and statistical analysis.
Investigate Which declared tolerances are attained by adaptive quadrature on a fixed family of endpoint-singular integrals?
2D elastodynamic metamaterials
UCI Machine Learning Repository
Pixelated metamaterial unit-cell designs and band-gap locations and widths for investigating simulation surrogate reliability.
Investigate How does a band-gap surrogate generalize to held-out metamaterial design regions?
3W dataset
UCI Machine Learning Repository
Oil-production process signals for studying event detection across operating scenarios.
Investigate How does event detection change when complete wells or production episodes are held out?
[Re] Easy Bayesian Transfer Learning with Informative Priors
ReScience C
A replication concerning Bayesian transfer learning with informative priors.
Investigate How sensitive are posterior predictive errors to a deliberately misspecified prior under a fixed transfer-data budget?
[Re] Fast-Activating Voltage- and Calcium-Dependent Potassium (BK) Conductance Promotes Bursting in Pituitary Cells: A Dynamic Clamp Study
ReScience
A replication of a pituitary-cell conductance model, with uncertainty quantification identified in source keywords.
Investigate Which bursting summary is robust to a prespecified parameter-uncertainty grid and solver refinement?
[Re] Measures for investigating the contextual modulation of information transmission
ReScience C
A replication concerning measures of contextual modulation in information transmission.
Investigate How do the selected measures behave on prespecified independence, redundancy and interaction examples?
[Re] Modeling Insect Phenology Using Ordinal Regression and Continuation Ratio Models
ReScience C
An R replication of insect phenology modeling with ordinal regression and continuation-ratio models.
Investigate How do ordinal probability estimates and calibration change across held-out time periods in a rights-cleared dataset?
[Re] Replication Study of DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks
ReScience C
A replication of causally aware synthetic-data generation aimed at fairness evaluation.
Investigate Can synthetic data retain chosen distributional properties while changing a prespecified fairness metric under a known causal model?
[Re] Resampling methods for evaluating classification accuracy of wildlife habitat models
ReScience C
A replication of resampling methods for evaluating wildlife-habitat classification models.
Investigate How do random and spatially structured resampling compare on synthetic data with controlled spatial dependence?
[Re] Spatial constraints underlying the retinal mosaics of two types of horizontal cells in cat and macaque
ReScience C
An R replication of spatial constraints in retinal-cell mosaics.
Investigate How do pairwise spatial summaries respond to known exclusion-distance and density settings?