Knockoffs with side information
Posted on (Update: )
This note is for Ren, Z., & Candès, E. (2023). Knockoffs with side information. The Annals of Applied Statistics, 17(2), 1152–1174.
\[\newcommand\sgn{\mathrm{sign}}\]Two classes of filters
focus on constructing a filter that satisifes Property 1 and also systematically uses all the available information to determine the ordering of hypotheses.
at each step $k$, the filter determines the ``least promising’’ hypothesis among the unexamined hypotheses based on the information in $\cF_k$
two types of filters that quantify ``least promising’’ in different ways
5.1 Predictive modeling
at step $k$, estimate the probability that the sign of a feature importance statistic is negative conditional on $\cF_k$
let $s_j = \sgn(W_j)$ and compute an estimate of $P(s_j = -1\mid \cF_k)$ for each remaining feature.
treat ${s_j}_{j=1}^p$ as the binary responses and the magnitude ${\vert W_j\vert}_{j=1}^p$ and side information ${U_j}_{j=1}^p$ as predictors
consider the model
\[g(P(s_j=-1\mid \vert W_j\vert, U_j)) = h(\vert W_j\vert, U_j)\]if one postulate a logistic regression model
\[h(\vert W_j\vert, U_j) = \beta_0 + \beta_1\vert W_j\vert + \beta_2^TU_j\]For a generalized additive model (GAM)
\[h(\vert W_j\vert, U_j) = \beta_0 + h_0(\vert W_j\vert) + h_1(U_{j1}) + \cdots + h_r(U_{jr})\]use ${s_{\pi_j}}_{j\le k}, {U_j}_{j\in [p]}, {\vert W_j\vert}_{j\in [p]}$ as training data to fit the chosen model and the fitted function $\hat h$ for predicting the signs of statistics among the unexamined hypotheses.
5.2 Bayesian modeling
not imposing any assumption on the data generating mechanism.
simply use Bayesian thinking for calculating the probability of a feature being nonnull and whether the Bayesian beliefs about features are true or not does not hurt FDR control.
- $S_j = 1$ means feature $j$ is a nonnull
- $S_j = 0$ means it is a null
where $\nu(\cdot)$ is a link function. Marginally,
\[W_j\mid S_j, U_j\sim \begin{cases} P_1(W_j\mid U_j)\\ P_0(W_j\mid U_j) \end{cases}\]quantify the possibility of a feature being null by inspecting the posterior probability $P(S_j = 0\mid \vert W_j\vert, U_j)$
the remaining task is to model $\cP_0, \cP_1$ and $\nu$
model the conditional law of $W_j$ via
\[p_s(w\mid u) = \delta_s\cdot 1_{w=0} + (1-\delta_s)\cdot 1_{w\neq 0}\cdot \frac{\beta_s(u)\cdot \exp(\beta_s(u)\cdot w)}{ (1+\exp(w))^{\beta_s(u)+1} }, \quad s = 0,1\]Let $\cN$ (resp., $\cB$) denote the class of function $\nu(\cdot)$ (resp., $\beta_0(\cdot), \beta_1(\cdot)$)
use EM to obtain the MLE
- GLM-extension approach: work with regularized log-likelihood
- Nonparametric regression approach: estimate the conditioal expectation instead of solving the optimization problems