AI evaluation
Choose inputs, comparisons and metrics before trusting a score.
Choose a useful starting point.
Two scores are comparable only when the task, input split, normalization, aggregation and failure handling are stated. Begin by freezing those choices and a simple baseline. The evaluation-harness brief asks whether two tools agree on the same saved predictions; it does not claim either tool has been run here.
For agent evaluations, preserve the environment, action trace, budget and unsuccessful attempts alongside the scoring rule. The linked protocols help you plan this record. They do not make a benchmark suitable for every agent or establish that a qualified runtime supports its dependencies.
Questions to work through.
Put uncertainty around a small agent-success benchmark
How misleading can a nominal 95% success-rate interval be when only twenty independent tasks are observed?
Explain a score difference before blaming the model
Do evaluation harnesses agree on the same frozen predictions once normalization, aggregation and failure handling are made explicit?
Make an agent benchmark attempt auditable
Can another operator reconstruct one bounded agent-evaluation attempt from its task, environment, actions, failures and scoring rule?
Explore the sources.
All 191 resources →Iris
UCI Machine Learning Repository
Flower measurements for a compact, interpretable multiclass baseline and leakage audit.
Investigate How much do iris classification results change when preprocessing is restricted to training folds?
3.1. Cross-validation: evaluating estimator performance
scikit-learn documentation
A guide to model-evaluation splits, including grouped and time-series data, nested model selection and permutation-based assessment.
Investigate How much optimism arises when repeated observations from the same group are split across train and test?
1.13. Feature selection
scikit-learn documentation
A guide to variance filters, univariate tests, recursive elimination, model-based selection and selection inside pipelines.
Investigate How much apparent accuracy is introduced by selecting features before cross-validation on noise-only data?
1.16. Probability calibration
scikit-learn documentation
A probability-calibration guide covering reliability curves and sigmoid, isotonic and temperature-scaling approaches.
Investigate How do calibration methods trade off log loss and reliability when the calibration sample is small?
2.7. Novelty and Outlier Detection
scikit-learn documentation
A guide distinguishing outlier detection from novelty detection and comparing assumptions of several anomaly-detection methods.
Investigate How does training-set contamination alter detection rates at a fixed false-positive target?
2D elastodynamic metamaterials
UCI Machine Learning Repository
Pixelated metamaterial unit-cell designs and band-gap locations and widths for investigating simulation surrogate reliability.
Investigate How does a band-gap surrogate generalize to held-out metamaterial design regions?
3.4. Metrics and scoring: quantifying the quality of predictions
scikit-learn documentation
A broad scoring reference that distinguishes classification, regression, ranking and clustering metrics and their scorer interfaces.
Investigate Can a scoring implementation reproduce independently computed confusion-matrix and probability-loss examples, including undefined edge cases?
3W dataset
UCI Machine Learning Repository
Oil-production process signals for studying event detection across operating scenarios.
Investigate How does event detection change when complete wells or production episodes are held out?
5.1. Partial Dependence and Individual Conditional Expectation plots
scikit-learn documentation
A guide to partial-dependence and individual-conditional-expectation curves, including definitions, computation and correlated-input caveats.
Investigate When features are correlated, how many evaluation points used by a response curve fall outside a declared support region?
5.2. Permutation feature importance
scikit-learn documentation
A model-inspection guide explaining permutation importance and its limitations when predictors are strongly correlated.
Investigate How does duplicating a predictive feature change individual permutation-importance scores while predictive accuracy stays comparable?
[Re] Badder Seeds: Reproducing the Evaluation of Lexical Methods for Bias Measurement
ReScience C
A replication examining lexical methods for measuring bias and sensitivity to seed choices.
Investigate How much does the selected bias metric vary across prespecified seed-set perturbations compared with its sampling uncertainty?
[Re] BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning
ReScience C
A replication of replay-based continual learning with vision transformers, relevant to evaluating retention across sequential tasks.
Investigate At a fixed replay-memory budget, how sensitive are retention measurements to task order?