WeiYa's Work Yard

A traveler with endless curiosity, who fell into the ocean of statistics, tries to write down his ideas and notes to save himself.

Joint Summarized by Marginal or Conditional?

Posted on
Tags: Marginal Distribution, Conditional Distribution, Hammersley-Clifford Theorem

I happened to read Yixuan’s blog about a question related to the course Statistical Inference, whether two marginal distributions can determine the joint distribution. The question is adopted from Exercise 4.47 of Casella and Berger (2002).

Let $X$ and $Y$ be independent $N(0,1)$ random variables, and define a new random variable $Z$ by $$ Z = \begin{cases} X & \text{ if }XY>0\\ -X & \text{ if }XY<0 \end{cases}\,. $$ It can be proved that $Z$ has a normal distribution, but the joint distribution of $Z$ and $Y$ is not bivariate normal by showing that $Z$ and $Y$ always have the same sign.

This exercise aims to illustrate that marginal normality does not imply bivariate normality.

This question recalled me related material in Gibbs sampler, where the conditional distributions contain sufficient information to produce a sample from the joint distribution. The Hammersley-Clifford theorem (Robert and Casella (2004)) demonstrates this feature,

The joint distribution associated with the conditional densities $f_{Y\mid X}(y\mid x)$ and $f_{X\mid Y}(x\mid y)$ has the joint density $$ f(x,y)=\frac{f_{Y\mid X}(y\mid x)}{\int [f_{Y\mid X}(y\mid x)/f_{X\mid Y}(x\mid y)]dy}\,. $$
Since $f(x,y)=f_{Y\mid X}(y\mid x)f_X(x)=f_{X\mid Y}(x\mid y)f_Y(y)$ we have $$ \int \frac{f_{Y\mid X}(y\mid x)}{f_{X\mid Y}(x\mid y)}dy = \int\frac{f_Y(y)}{f_X(x)}dy=\frac{1}{f(X)}\,, $$ and the result follows.

In a word, the full conditional distributions perfectly summarize the joint density, but the set of marginal distributions obviously fails to do so.

It is necessary to note that this theorem makes the implicit assumption that the joint density $f(x,y)$ exists. The assumption can be not satisfied (Example 10.29 of Robert and Casella (2004)), suppose a pair of conditional densities

\[X_1\mid x_2\sim E(x_2),\qquad X_2\mid x_1\sim E(x_1)\]

are well defined, but

\[\int \frac{f(x_1\mid x_2)}{f(x_2\mid x_1)} = \infty\,,\]

which violates the requirement of the Hammersley-Clifford theorem.

Published in categories Report 
← previous    next →

See all posts →