Test an analysis against data with known generating truth
Can the analysis distinguish finite-sample variation from leakage, missingness bias, or a covariance bug?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
Make the question your own.
5 sources · Exact preparation context
- Synthetic regression data with declared coefficients and split →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: 8b9f83fdbabd9883dfbbd0dc610582b2f75f9383d789035b8588fb9bf89674b0
- Synthetic missingness data with complete ground truth →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: 9390d4720ae6b1abd1d161dc20aa599ec55d41c5d6eda9c60c7a6d10e6564a27
- Synthetic correlated measurements with a known covariance →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: 0b97e4906530203de2d02d0ba8cbbe61ca9cbe5c677c4d92146753f1c715b133
- 3.1. Cross-validation: evaluating estimator performance →
scikit-learn documentation. 3.1. Cross-validation: evaluating estimator performance. https://scikit-learn.org/stable/modules/cross_validation.html; observed version 1.9.1 (observed documentation version; stable URL is mutable).
Record revision: 8c31ebc272585f1e00b12cc000e755dfa687ab3dba882effad7dd1a7d5ea149a
- 3.4. Metrics and scoring: quantifying the quality of predictions →
scikit-learn documentation. 3.4. Metrics and scoring: quantifying the quality of predictions. https://scikit-learn.org/stable/modules/model_evaluation.html; observed version 1.9.1 (observed documentation version; stable URL is mutable).
Record revision: 14c631ef986b838e43bdf04a5e3fd3a3be03cb58771e82dd8b05cc39b4aa6282
Source links open the current listing. This plan retains the revisions above; listing review does not grant permission to copy or adapt source files.
Before carrying out this plan
- Three original generators are prepared for the qualified standard-library runtime.
- Scikit-learn comparisons require a separately qualified environment.
Limits to keep in view
- Synthetic rows are not real-world measurements.
- No held-out score for a named external model has been measured.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.
What to compare
Use the fixed intercept/slope/noise model, the declared normal missingness mechanism, and the known covariance matrix. Do not require a finite realization to match population moments exactly.
Useful outputs
- Generated rows, model parameters, seeds, masks and a frozen split.
- Training-only fit/preprocessing and separate held-out metrics.
- Population comparison quantities alongside finite-sample observations.
Inputs and prerequisites
- Three original generators are prepared for the qualified standard-library runtime.
- Scikit-learn comparisons require a separately qualified environment.
What this work would not establish
- Synthetic rows are not real-world measurements.
- No held-out score for a named external model has been measured.
Start with these sources.
Synthetic regression data with declared coefficients and split
Executable Science local pilot drafts
Generate 128 regression observations with known coefficients, a declared Gaussian noise process, and a fixed train/test split, together with a small reference fit.
Investigate Can another implementation recover the reference fit and evaluate predictions without changing the test split?
Synthetic missingness data with complete ground truth
Executable Science local pilot drafts
Generate paired complete, independently missing, and value-dependent missing observations from a declared standard-normal process.
Investigate How does a known missingness mechanism affect a complete-case mean in this controlled example?
Synthetic correlated measurements with a known covariance
Executable Science local pilot drafts
Generate 256 paired synthetic measurements through a declared linear transformation of independent standard-normal draws, with known population covariance.
Investigate Can covariance and correlation implementations be calibrated against a transparent generator?
3.1. Cross-validation: evaluating estimator performance
scikit-learn documentation
A guide to model-evaluation splits, including grouped and time-series data, nested model selection and permutation-based assessment.
Investigate How much optimism arises when repeated observations from the same group are split across train and test?
3.4. Metrics and scoring: quantifying the quality of predictions
scikit-learn documentation
A broad scoring reference that distinguishes classification, regression, ranking and clustering metrics and their scorer interfaces.
Investigate Can a scoring implementation reproduce independently computed confusion-matrix and probability-loss examples, including undefined edge cases?