Asymptotic Optimal Knockoffs via MLR
Posted on
\[\newcommand\MLR{\mathrm{MLR}}\]- for knockoff, it is not clear which statistic maximizes power.
- it argues that state-of-the-art lasso-based feature statistics often prioritize features that are unlikely to be discovered, leading to lower power in real applications
- introduce masked likelihood ratio statistics, which prioritize features according to one’s ability to distinguish each feature from its knockoff.
- although no single feature statistic is uniformly most powerful in all situations, MLR statistics asymptotically maximize the number of discoveries under a user-specified Bayesian control of the data.
Introduction
- analyzing the power of knockoffs can be very challenging
1.2 Theoretical problem statement
define two types of optimal knockoff statistics:
Let $S_w \subset [p]$ denote the discovery set using feature statistic $w$ on data $([X, \tilde X, Y])$
- oracle statistics: maximize the expected number of discoveries (ENDisc) for the true (unknown) data distribution $P^\star$
- bayes-optimal statistics: maximize ENDisc with respect to a prior distribution over $P^\star$
let $\cP={P^{(\theta)}:\theta\in \Theta}$ denote a model class of potential distributions for $(X, Y)$ and let $\pi$ denote a prior density over $\cP$
\[\text{ENDisc}^\pi(w) = \int_\Theta\bbE_{P^{(\theta)}}[\vert S_w\vert]\pi(\theta)d\theta = \bbE_{P^\pi}[\vert S_w\vert]\]- $P^\pi$: denotes the mixture distribution which first samples a parameter $\theta^\star\in \Theta$ according to the prior $\pi$ and then sample $(X, Y)\mid \theta^\star \sim P^{(\theta^\star)}$
1.3 Contribution and overview of results
for any value $d = (y, {x_j, \tilde x_j}_{j=1}^p)$ in the support of $D$ and any fixed $x\in {x_j, \tilde x_j}$, let
\[P_j^\pi(x\mid d) = P^\pi(X_j=x\mid D=d)\]denote the conditional law of $X_j$ given $D$.
the masked likelihood ratio (MLR) statistic:
\[\text{MLR}_j^\pi = \log\left(\frac{P_j^\pi(X_j\mid D)}{P_j^\pi(\tilde X_j\mid D)}\right)\]Theory: asymptotic Bayes-optimality
MLR statistics asymptotically maximize the number of expected discoveries under $P^\pi$
- do not assume that $Y\mid X$ follow a linear model under $P^\pi$
Empirical results
- concrete instantiations of MLR statistics based on uninformative (sparse) priors for generalized additive models and binary GLMs
2 Intuition and motivation from an HIV drug resistance dataset
3 Masked likelihood ratio statistics
3.1 Knockoffs as inference on masked data
to maximize power, we should assign $W_j$ a large absolute value iff $P^\star(W_j > 0)$ is large
Suppose we observe data $X, Y$, knockoffs $\tilde X$, and independent random noise $U$. The masked data is defined as
\[D = \begin{cases} (Y, \{X_j, \tilde X_j\}_{j=1}^p, U) & \text{ for model-X knockoffs}\\ (X, \tilde X, \{X_j^TY, \tilde X_j^TY\}_{j=1}^p, U) & \text{for fixed-X knockoffs} \end{cases}\]reformulate knockoffs as a guessing game, where we produce a guess $\hat X_j\in {X_j, \tilde X_j}$ of the value of $X_j$ based on $D=(Y, {X_j, \tilde X_j}_{j=1}^p)$
if our guess is right, meaning $\hat X_j=X_j$, then we are rewarded and $W_j > 0$: else $W_j < 0$.
only assign $W_j$ a large absolute value when we are confident that our guess $\hat X_j$ is correct.
3.2 Introducing MLR statistics
\[\MLR_j^\pi = \log\left( \frac{ \int_\Theta P_j^{(\theta)}(X_j\mid D)\pi(\theta\mid D)d\theta }{ \int_\Theta P_j^{(\theta)}(\tilde X_j\mid D)\pi(\theta\mid D)d\theta } \right)\]for any other feature statistic $W$,
\(P^\pi(\MLR_j^\pi > 0\mid D) \ge P^\pi(W_j > 0\mid D)\) Furthermore, \(P^\pi(\MLR_j^\pi > 0\mid D) = \frac{\exp(\vert\MLR_j^\pi\vert)}{1+\exp(\vert \MLR_j^\pi\vert)}\)
- Ren and Cand`es (2020): also suggest ranking the hypotheses by $P(W_j> 0\mid \vert W_j\vert)$. develop “adaptive knockoffs”, an extension of knockoffs that can be combined with any predefined feature statistic, including MLR or lasso statistics.
- Katsevich and Ramdas (2020): the unmasked likelihood statistic maximizes $P^\star(W_j > 0)$. unlike MLR statistics, the unmasked likelihood statistic is not a valid knockoff statistic even though it is marginally symmetric under the null
3.3 MLR statistics are asymptotically optimal
Theorem 3.2 shows that MLR statistics asymptotically maximize the normalized number of expected discoveries without any explicit assumptions on the relationship between $Y$ and $X$ or the dimensionality.
3.4 Maximizing the expected number of true discoveries
adjusted MLR (AMLR), which asymptotically maximize the number of expected true discoveries under $P^\pi$.
AMLR and MLR statistics are different but not too different.
AMLR asymptotically maximize power under $P^\pi$.
4 Computing MLR statistics
4.1 General strategy
the conditional likelihood ratio equals
\[\frac{P^\pi(X_j=x_j\mid X_{-j}, \theta^\star = \theta, D=d)}{P^\pi(X_j=\tilde x_j\mid X_{-j}, \theta^\star = \theta, D=d)} = \frac{P_{Y\mid X}^{(\theta)}(y\mid X_j = x_j, X_{-j})}{P_{Y\mid X}^{(\theta)}(y\mid X_j=\tilde X_j, X_{-j})}\]
4.2 A default choice of Bayesian model
- Example 1: sparse generalized additive model