Design a generative-model calibration check
Can inference recover declared generating quantities without conflating posterior predictive fit and parameter calibration?
Make this plan your own ↓Read, edit and export without an account. This is preparation; no run or result is claimed.
What to compare
Distinguish simulation-based calibration, posterior predictive checks and one-chain occupation frequencies. A different implementation under the same operator does not establish independent verification.
Useful outputs
- A model, prior/generator, inference configuration and frozen calibration statistic.
- A simulation-budget and uncertainty plan that accounts for dependent samples.
- A mapping from diagnostic observations to the narrow claims they can support.
Inputs and prerequisites
- Read the source methods and retain the separate documentation/code license scopes.
- Stan/PyMC/ArviZ/deep-model environments are outside the present standard-library qualification.
What this work would not establish
- The cited replication remains an external publication.
- Stan documentation includes a no-derivatives license observation; link and describe within the recorded scope.
Start with these sources.
Synthetic correlated measurements with a known covariance
Generate 256 paired synthetic measurements through a declared linear transformation of independent standard-normal draws, with known population covariance.
Stationary-distribution calibration for a two-state chain
Compare a known stationary distribution with matrix iteration and occupation frequencies from a deterministic-seed Markov-chain simulation.
Simulation-Based Calibration Checking
A guide to checking Bayesian inference algorithms by simulating parameters and datasets from a model and examining posterior ranks or coverage.
Posterior and Prior Predictive Checks
A guide to prior and posterior predictive checks that compare replicated-data summaries with chosen aspects of observed data.
PyMC
Probabilistic modeling and Bayesian inference with automatic sampling methods.
ArviZ
Diagnostics, summaries, and visualization support for Bayesian inference results.
[Re] Improved Calibration and Predictive Uncertainty for Deep Neural Networks
A replication about calibration and predictive uncertainty in deep neural networks, including mixup in its source keywords.
Make the question your own.
Edit freely. Export a copy before reloading or leaving. Your text stays in this tab until you choose to copy, download or save it privately.