Approximating Bayes

Posted on Jul 04, 2024 (Update: Aug 16, 2024) 0 Comments

Tags: Variational Bayes, Approximate Bayesian Computation

This is the note for Martin, G. M., Frazier, D. T., & Robert, C. P. (2024). Approximating Bayes in the 21st Century. Statistical Science, 39(1), 20–45. https://doi.org/10.1214/22-STS875

approximate Bayes methods:

• produce computational solutions to certain “intractable” statistical problems that challenge exact methods like MCMC

four main approximate techniques:

1. approximate Bayesian computation (ABC)
2. Bayesian synthetic likelihood (BSL)
3. variational Bayes (VB)
4. integrated nested Laplace approximation (INLA)

simulation-based approaches: ABC and BSL optimization-based approaches: VB and INLA

Bayesian computation in a nutshell

• data generating process: $p(y\mid \theta)$
• prior belief: $p(\theta)$

\[p(\theta\mid y) \propto p(y\mid \theta)p(\theta)\]

the quantities that underpin the whole of Bayesian analysis can be expressed as expectations

\[\bbE[g(\theta)\mid y]\]

and

\[\bbE[g(\theta)\mid \cM]\]

all Bayesian computational techniques:

• deterministic integratiomn methods
• exact simulation methods
• approximate methods

Simulation-based approaches

approximate bayesian computation (ABC)

in cases where, despite the complexity of the problem preventing the evaluation of $p(y\mid \theta)$, $p(y\mid\theta)$ and $p(\theta)$ can still be simulated from

Bayesian synthetic likelihood (BSL)

Optimization-based approaches

Variational Bayes (VB)

Integrated nested Laplace approximation

Laplace asymptotic approximation

Hybrid approximate methods