NGBoost: Natural Gradient Boosting
Posted on
NGBoost: Natural Gradient Boosting for Probabilistic Prediction
- NGBoost generalizes gradient boosting to probabilistic regression by treating the parameters of the conditional distribution as targets for a multiparameter boosting algorithm.
Introduction
- estimate the conditional probability distribution $P(y\mid x)$ for each value of $x$ instead of producing a point estimate like $E[y\mid x]$. This is called probabilistic regression
- probabilistic estimation is already the norm in classification problems
- any mean-estimating regression method can be made probabilistic by assuming homoscedasticity and estimating an unconditional noise model
- Generalized Additive Models for Shape, Scale, and Location (GAMLSS) allow heteroscedasticity but are restricted to a pre-specified method form
- Bayesian methods naturally generate predictive uncertainty estimates by integrating predictions over the posterior
- GBM are a set of highly-modular methods that work out-of-the-box and perform well on structured input data
- in regression tasks, they output only a scalar value. under a squared-error loss these scalars can be interpreted as the mean of a conditional Gaussian distribution with some (unknown) constant variance
- such probabilistic interpretations have little use if the variance is assumed constant
- the predicted distributions need to have at least two degrees of freedom to convey both the magnitude and the certainty of the predictions
Natural Gradient Boosting
interested in producing a probability distribution $P_\theta(y\mid x)$ (with CDF $F_\theta$)
the approach is to assume $P_\theta(y\mid x)$ is of a specified parameteric form, then estimate the $p$ parameters $\theta\in \IR^p$ of the distribution as functions of $x$
3.1 Proper Scoring Rules
in probabilistic regression is a scoring rule, which compares the estimated probability distribution to the observed data
in mathematical notation, a scoring rule $S$ is a proper scoring rule iff it satisifies
\[\bbE_{y\sim Q}[S(Q, y)] \le \bbE_{y\sim Q}[S(P, y)]\qquad \forall P, Q\]where $Q$ represents the true distribution of outcomes $y$, and $P$ is any other distribution
the most commonly used proper scoring rule is the logarithmic score $\cL$, which, when minimized, givens the MLE
\[\cL(\theta, y) = -\log P_\theta(y)\]another example is CRPS, which is a robust alternative to MLE
The CRPS is defined as
\[C(\theta, y) = \int_{-\infty}^y F_\theta(z)^2dz + \int_y^\infty (1-F_\theta(z))^2dz\]3.2 The Generalized Natural Gradient
\[\Delta S(\theta, y)\propto\lim_{\epsilon \rightarrow 0}\argmax_{d:\Vert d\Vert = \epsilon} \cS(\theta+d, y)\]this gradient is not invariant to reparameterization
the problem is that ``distance’’ between two parameter values does not correspond to an appropriate distance between the distributions that those parameters identify
this motivates the natural gradient (\tilde\Delta), which originated in information geometry
every proper scoring rule induces a divergence that can serve as local distance metric in the space of distributions
\[D_\cS(Q\Vert P) = \bbE_{y\sim Q}[\cS(P, y)] - \bbE_{y\sim Q}[\cS(Q, y)]\]the MLE scoring rule induces the KL divergence
while CRPS induces the $L^2$ divergence
the divergences $D_{KL}$ and $D_{L^2}$ are invariant to how $Q$ and $P$ are parameterized
Natural Gradient
the generalized natural gradient is the direction of steepest ascent in Riemannian space, which is invariant to parameterization, and is defined as
\[\tilde\nabla \cS(\theta, y)\propto \lim_{\epsilon\rightarrow 0} \argmax_{d: D_\cS(P_\theta\Vert P_{\theta+d}) = \epsilon} \cS(\theta+d, y)\]the natural gradient of the form
\[\tilde \nabla \cS(\theta, y) \propto I_S(thtea)^{-1}\nabla \cS(\theta, y)\]using the natural gradient for learning the parameters makes the optimization problem invariant to parameterization and leads to more efficient and stable learning dynamics.
3.4 NGBoost: Natural Gradient Boosting
the NGBoost is a supervised learning method for probabilistic prediction that uses boosting to estimate the parameters of a conditional probability distribution $P(y\mid x)$ as functions of $x$
Analysis and Discussion
- when the probability distribution is in the exponential family and the choice of parameterization is the natural parameters of that family, then a Newton-Raphson step is equivalent to a natural gradient descent step.