Design a generative-model calibration check
Can inference recover declared generating quantities without conflating posterior predictive fit and parameter calibration?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
Make the question your own.
7 sources · Exact preparation context
- Synthetic correlated measurements with a known covariance →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: 0b97e4906530203de2d02d0ba8cbbe61ca9cbe5c677c4d92146753f1c715b133
- Stationary-distribution calibration for a two-state chain →
Original local preparation by the Executable Science seed collection; upstream API references remain separately attributed.
Record revision: d70cacc9234ff3d29fe582cd89034f3f2b7ad79b3048c243b36626110dc5db53
- Simulation-Based Calibration Checking →
Stan User's Guide / Stan Development Team. Simulation-Based Calibration Checking. https://raw.githubusercontent.com/stan-dev/docs/1bc688a6f9a6c6931b7527c333345d2dcf6ee0bb/src/stan-users-guide/simulation-based-calibration.qmd; observed version 1bc688a6f9a6c6931b7527c333345d2dcf6ee0bb.
Record revision: a35215e18179ebff96817ce04aa387c1c91f1912ba88d395921402c0ddca323e
- Posterior and Prior Predictive Checks →
Stan User's Guide / Stan Development Team. Posterior and Prior Predictive Checks. https://raw.githubusercontent.com/stan-dev/docs/1bc688a6f9a6c6931b7527c333345d2dcf6ee0bb/src/stan-users-guide/posterior-predictive-checks.qmd; observed version 1bc688a6f9a6c6931b7527c333345d2dcf6ee0bb.
Record revision: 67b570638e2413a32eba55407a237e9cf4ae5975505cea7586253f597a12dfcc
- PyMC →
Preserve the upstream project name, version, repository link, applicable notices, and contributor attribution when preparing an artifact for reuse.
Record revision: 98b54e4e553655239feae686b3146d9a4bceccfa06b2ac185e882bc961bc659b
- ArviZ →
Preserve the upstream project name, version, repository link, applicable notices, and contributor attribution when preparing an artifact for reuse.
Record revision: 4d908b2d6e7b0b192be6b057f0ff6009a3cf267cd0702208e8318d538a4369cd
- [Re] Improved Calibration and Predictive Uncertainty for Deep Neural Networks →
Singh, Aditya; Bay, Alessandro. [Re] Improved Calibration and Predictive Uncertainty for Deep Neural Networks. ReScience C 6(2), #10; 10.5281/zenodo.3818605.
Record revision: 0964837b93e8b349676fc61ba22e611cf8ac0919f4ee016a51e04e8a56990391
Source links open the current listing. This plan retains the revisions above; listing review does not grant permission to copy or adapt source files.
Before carrying out this plan
- Read the source methods and retain the separate documentation/code license scopes.
- Stan/PyMC/ArviZ/deep-model environments are outside the present standard-library qualification.
Limits to keep in view
- The cited replication remains an external publication.
- Stan documentation includes a no-derivatives license observation; link and describe within the recorded scope.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.
What to compare
Distinguish simulation-based calibration, posterior predictive checks and one-chain occupation frequencies. A different implementation under the same operator does not establish independent verification.
Useful outputs
- A model, prior/generator, inference configuration and frozen calibration statistic.
- A simulation-budget and uncertainty plan that accounts for dependent samples.
- A mapping from diagnostic observations to the narrow claims they can support.
Inputs and prerequisites
- Read the source methods and retain the separate documentation/code license scopes.
- Stan/PyMC/ArviZ/deep-model environments are outside the present standard-library qualification.
What this work would not establish
- The cited replication remains an external publication.
- Stan documentation includes a no-derivatives license observation; link and describe within the recorded scope.
Start with these sources.
Synthetic correlated measurements with a known covariance
Executable Science local pilot drafts
Generate 256 paired synthetic measurements through a declared linear transformation of independent standard-normal draws, with known population covariance.
Investigate Can covariance and correlation implementations be calibrated against a transparent generator?
Stationary-distribution calibration for a two-state chain
Executable Science local pilot drafts
Compare a known stationary distribution with matrix iteration and occupation frequencies from a deterministic-seed Markov-chain simulation.
Investigate Do the two computational routes approach the same analytical stationary distribution?
Simulation-Based Calibration Checking
Stan User's Guide / Stan Development Team
A guide to checking Bayesian inference algorithms by simulating parameters and datasets from a model and examining posterior ranks or coverage.
Investigate Can a deliberately biased posterior sampler be distinguished from an exact conjugate sampler by a prespecified rank-uniformity diagnostic?
Posterior and Prior Predictive Checks
Stan User's Guide / Stan Development Team
A guide to prior and posterior predictive checks that compare replicated-data summaries with chosen aspects of observed data.
Investigate Which discrepancies detect a known tail-shape mismatch that a mean-only predictive check misses?
PyMC
Upstream project repository
Probabilistic modeling and Bayesian inference with automatic sampling methods.
Investigate Do posterior samples from a simple conjugate model agree with its analytic posterior moments and quantiles?
ArviZ
Upstream project repository
Diagnostics, summaries, and visualization support for Bayesian inference results.
Investigate Which diagnostics distinguish deliberately separated chains from independently generated draws of the same target?
[Re] Improved Calibration and Predictive Uncertainty for Deep Neural Networks
ReScience C
A replication about calibration and predictive uncertainty in deep neural networks, including mixup in its source keywords.
Investigate How do log loss, calibration summaries and selective-prediction error compare under a fixed validation and test split?