WeiYa's Work Yard

A dog, who fell into the ocean of statistics, tries to write down his ideas and notes to save himself.

Frequentist Accuracy of Bayesian Estimates

Posted on
Tags: Delta Method, Bayesian Inference

This note is for Efron’s slide: Frequentist Accuracy of Bayesian Estimates, which is recommended by Larry’s post: Shaking the Bayesian Machine.

In Bayesian Inference, we have

  • Parameter: $\mu\in \Omega$
  • Observed data: $x$
  • Prior: $\pi(\mu)$
  • Probability distributions: $\{f_\mu(x),\mu\in\Omega\}$
  • Parameter of interest: $\theta = t(\mu)$

then the posterior estimate of $\theta$ is

\[\E[\theta\mid x] = \frac{\int_\Omega t(\mu)f_\mu\pi(\mu)d\mu}{\int_\Omega f_\mu(x)\pi(\mu)d\mu}\,.\]

This form recalls me a more generic form of the conditional expectation.

The usual uninformative prior is Jeffreys’ prior,

\[\pi(\mu) = \vert I(\mu)\vert^{1/2}\,,\]

where $I(\mu) = \Cov[\nabla_\mu\log f_\mu(x)]$.

But how accurate are the estimates? Then we will consider the frequentist variability of $\E[t(\mu)\mid x]$.

General Accuracy Formula

  • $\mu$ and $x\in\bbR^p$
  • $x\sim (\mu, V_\mu)$ (here I guess $\mu$ is not necessarily the expectation of $x$ according to the following examples, this symbol might want to indicate $x$ is indexed by $\mu$)
  • $\alpha_x(\mu)=\nabla_x\log f_\mu(x)$
$\hat E=E[t(\mu)\mid x]$ has gradient $$ \nabla_xE =\Cov[t(\mu), \alpha_x(\mu)\mid x]\,. $$
The delta-method standard deviation of $E$ is $$ \sd(\hat E) = \left\{\Cov[t(\mu),\alpha_x(\mu)\mid x]'V_x\Cov[t(\mu),\alpha_x(\mu)\mid x]\right\}^{1/2}\,. $$


Suppose we have the posterior sample $\mu_1^*,\ldots,\mu_B^*$, let $t_i^*=t(\mu_i^*)$ and $\alpha_i^*=\alpha_x(\mu_i^*)$, then

\[\widehat{\cov} = \sum_{i=1}^B(\alpha_i^*-\bar \alpha)(t_i^*-\bar t)/B\,,\]

it follows that

\[\hat \sd(\hat E) = \left[\widehat\cov'V_x\widehat\cov\right]^{1/2}\,.\]

Published in categories Note