Explain a score difference before blaming the model
Do evaluation harnesses agree on the same frozen predictions once normalization, aggregation and failure handling are made explicit?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
Make the question your own.
6 sources · Exact preparation context
- Inspect AI →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 22506ea43ddc13005743337266a736e5e51d2754dcfb98771ea499aaf2cc3ad3
- Language Model Evaluation Harness →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: a2c1fada2bde773d2a536dc4593216155a84ba7d2ce5e72478638a9e4f676fd4
- LightEval →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: e4b84f590f3c177d5fabd1b588f943c2ba0b0bcfe0ef6beef819b25ec5418154
- TorchMetrics →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 1487479c180cab201746832380b127d794b3fca6cb998b140b1820bad0e10183
- Uncertainty Toolbox →
Preserve the upstream project name, repository, exact source commit and applicable contributor/notices; resolve upstream citation guidance for any later formal use.
Record revision: 49a5a0bcc644c3f02b42d3e1280e3c514af8e49606689e360f68c9a7f5d174f1
- Exact binomial coverage of Wald and Wilson intervals →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: c3675952145f7c18cb07be1d98419d0aac99b5238319a042b7f7fe157f88389b
Source links open the current listing. This plan retains the revisions above; listing review does not grant permission to copy or adapt source files.
Before carrying out this plan
- Read the pinned APIs and prepare adapters against a qualified dependency environment.
- Benchmark data, model weights, provider terms and API spending are separate from harness code licenses.
- The current qualified runtime does not include these external harnesses.
Limits to keep in view
- This is an unrun protocol proposal; it reports no harness discrepancy or model ranking.
- The referenced uncertainty tools do not automatically validate every benchmark design.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.
What to compare
Hold predictions fixed while comparing scorers. Match denominators and case inclusion before comparing aggregate scores. Model calls are a later, separately budgeted step; distinguish task dependence from an IID binomial model.
Useful outputs
- A small original prediction/answer fixture with exact expected scores and explicit failure rows.
- A mapping of prompt/template, scorer, normalization, aggregation, truncation and cache conventions.
- A source-version-pinned comparison report that separates fixture agreement from model performance.
Inputs and prerequisites
- Read the pinned APIs and prepare adapters against a qualified dependency environment.
- Benchmark data, model weights, provider terms and API spending are separate from harness code licenses.
- The current qualified runtime does not include these external harnesses.
What this work would not establish
- This is an unrun protocol proposal; it reports no harness discrepancy or model ranking.
- The referenced uncertainty tools do not automatically validate every benchmark design.
Start with these sources.
Inspect AI
Upstream project repository
Evaluation framework for composing tasks, model interactions, scoring and inspectable logs.
Investigate Does a deterministic mock-model evaluation preserve the input, scoring rule and retry history needed to audit every result?
Language Model Evaluation Harness
Upstream project repository
Task and model interfaces for reproducible language-model evaluation with inspectable prompting and scoring choices.
Investigate Can a tiny authored multiple-choice task expose the effect of length normalization on the selected answer?
LightEval
Upstream project repository
Language-model evaluation tooling for tracking prompt construction, scoring choices and evaluation configuration.
Investigate Can two declared scoring configurations be compared without changing task examples or model revision?
TorchMetrics
Upstream project repository
Metrics for tensor-based learning systems, useful for testing reductions, state accumulation and distributed aggregation.
Investigate Does a selected classification metric match a hand-built confusion matrix across uneven batch partitions?
Uncertainty Toolbox
Upstream project repository
Predictive-uncertainty analysis tools for examining calibration, sharpness and scoring on controlled regression predictions.
Investigate Can a fixed synthetic prediction set distinguish calibration changes from changes in interval width?
Exact binomial coverage of Wald and Wilson intervals
Executable Science local pilot drafts
Enumerate all binomial outcomes at n=20 and p=0.05, then compare exact interval-coverage calculations with a fixed-seed simulation.
Investigate Do nominal 95% intervals achieve that coverage in this prespecified small-sample setting?