Estimate FDR of Variable Selection
Posted on (Update: )
This note is for Luo, Yixiang, William Fithian, and Lihua Lei. “Estimating the FDR of Variable Selection.” arXiv:2408.07231. Preprint, arXiv, August 17, 2024.
\[\newcommand\FDR{\mathrm{FDR}}\]1.1 FDR estimation in the Gaussian linear model
\[Y = X\theta + \varepsilon, \varepsilon N(0, \sigma^2I_n)\]we can linearly decompose the FDR as
\[\FDR = \sum_{j=1}^d \FDR_j, \quad \FDR_j = \bbE\left[\frac{1\{j\in\cR\}}{\vert \cR\vert}\right]\cdot 1\{j\in \cH_0\}\]using this decomposition, a promising estimator of the FDR is
\[\widehat\FDR = \sum_{j=1}^d\widehat\FDR_j, \widehat\FDR_j = \bbE_{H_j}\left[\frac{1\{j\in \cR\}}{\vert \cR\vert}\mid S_j\right]\cdot \frac{1\{p_j > 0.1\}}{0.9}\]where $p_j$ is the p-value for the usual two-sided t-test, and $S_j = (X_{-j}^TY, \Vert Y\Vert^2)$ is a sufficient statistic under the null model $H_j$.
under $H_j$, the first factor in $\widehat\FDR_j$ is the UMVUE of $\FDR_j$, obtained by Rao-Blackwellization,
2 General formulation of the estimator
2.2 FDR estimation
Step 1: Estimating the first factor
Step 2: Estimating the second factor
one could obtain a conservative estimator by simply upper-bounding the indicator $1{j\in\cH_0}\le 1$, resulting in the estimator
\[\widehat\FDR^\star = \sum_j\widehat\FDR_j^\star\]a better estimator of the indicator should reliably eliminate terms for strong signal variables without introducing bias in terms for noise variables.
for any estimator $\psi_j(D)\in [0, 1]$, define its normalized version:
\[\phi_j(D) = \frac{\psi_j(D)}{\bbE_{H_j}[\psi_j(D)\mid S_j]}\]we have $\bbE_{H_j}[\phi_j\mid S_j] = 1$ almost surely.
take
\[\phi_j(p_j) = \frac{1\{p_j>\zeta\}}{1-\zeta}\]For any selection procedure $\cR$, and for any estimator $\psi_j:D\rightarrow [0, 1]$ of $1{j\in \cH_0}$, the estimator has non-negative bias: \(\bbE[\widehat\FDR] \ge \FDR\)
-
remark 2.3. the sole statistical assumption — availability of a sufficient statistic $S_j$ for the submodel defined by $H_j$ — is formally vacuous, because $D$ is itself a sufficient statistic. However, if we take $S_j = D$, we have \(\widehat\FDR_j(D) = \frac{1\{j\in \cR\}}{R}\) and consequently $\widehat\FDR=1$ almost surely.
-
remark 2.4. for variable selection by thresholding independent $p$-values $p_j$ at threshold $c\in (0, 1)$, Storey (2002) proposed a conservative biased FDR estimator
2.3 Understanding the method
consider the per-family error rate (PFER)
\[\text{PFER} = \bbE[\vert\cR\cap\cH_0\vert] = \sum_{j=1}^d P[j\in \cR]\cdot 1\{j\in \cH_0\} = \sum_{j\in \cH_0} P[j\in \cR]\]in problems where the number $R$ of selections is relatively large and stable, such that $1/R$ is roughly constant conditional on $S_j$, we have
\[\widehat\FDR = \frac{\widehat{PFER}}{R}\]- remark 2.5. The PFER is interesting in its own right as an upper bound for the family-wise error rate (FWER)
2.4 Example model assumptions
Example 2.1: Nonparameteric model-X setting
assume that one observe $n$ independent realizations of the $d+1$-variate random vector $(X, Y)\sim P$, where the joint distribution $P_X$ of expmanatory variables is fully known, but nothing is assumed about the conditional distribution $P_{Y\mid X}$
Example 2.2: Gaussian graphical model
a complete sufficient statistic for $H_{jk}$ is given by observing every entry of $\hat\Sigma$ except $\hat\Sigma_{jk}$ and $\hat\Sigma_{kj}$, or equivalently by
\[S_{jk} = (X_{-k}^TX_{-k}, X^T_{-\{j, k\}}X_k, \Vert X_k\Vert^2)\]3 Standard error of $\widehat{\FDR}$
3.1 Theoretical bound for standard error
- Assumption 3.1: block-orthogonal variables
-
Condition 3.1: enough selections
-
Theorem 3.1: finite sample variance bound
- Corollary 3.1: Asymptotically vanishing variance
3.2 Standard error estimation by bootstrap
4 Real world examples
4.1 HIV drug resistance studies
4.2 Protein-signaling network
5 Simulation studies
5.1 Gaussian linear model
iid normal:
- weaker signals: potentially lead to significant bias
- dense signals: lead to a higher bias because signal variables’ contribution is the source of bias
- aspect ratio close to 1: closer to collinear. this could lead to higher bias because the p-value used in the bias correction are weak.
- very sparse: lead to high variance in FDP and potentially also in $\widehat{\FDR}$