[Paper Review] Learning Exponential Families in High-Dimensions: Strong Convexity and Sparsity

Sham M. Kakade, Ohad Shamir+2 more|arXiv (Cornell University)|Oct 31, 2009

Sparse and Compressive Sensing Techniques9 references22 citations

TL;DR

This paper establishes a strong convexity condition for general exponential families that enables non-asymptotic analysis of $L_1$-regularized estimation in high-dimensional settings. It shows that under mild moment growth conditions on sufficient statistics, the prediction loss behaves as a strongly convex function once a quantified 'burn-in' sample size is reached, leading to an $O(s \log p / n)$ convergence rate for both prediction loss and estimation error, with a two-stage procedure achieving a sparse model of size $O(s)$.

ABSTRACT

The versatility of exponential families, along with their attendant convexity properties, make them a popular and effective statistical model. A central issue is learning these models in high-dimensions, such as when there is some sparsity pattern of the optimal parameter. This work characterizes a certain strong convexity property of general exponential families, which allow their generalization ability to be quantified. In particular, we show how this property can be used to analyze generic exponential families under L_1 regularization.

Motivation & Objective

• To characterize the strong convexity properties of general exponential families in high-dimensional settings where $p \gg n$.
• To quantify the sample size required for the prediction loss to behave as a strongly convex function, analogous to a 'burn-in' phase in optimization.
• To extend $L_1$-regularization analysis beyond linear models to general exponential families, establishing convergence rates for prediction and estimation error.
• To develop a two-stage procedure that yields a sparse model with $O(s)$ non-zero features and near-optimal risk.
• To relate the convergence rate to intrinsic properties of the exponential family, such as standardized moments and Fisher information.

Proposed method

• Introduces a growth condition on standardized moments and cumulants of sufficient statistics, allowing quantification of strong convexity in the prediction loss.
• Uses sub-Gaussian concentration bounds for sufficient statistics to control the deviation of empirical expectations from their true values.
• Applies a two-stage estimation procedure: first estimate with $L_1$ regularization, then refit on the set of coordinates with large estimated coefficients.
• Employs the Restricted Eigenvalue (RE) condition on the design matrix to ensure sparse recovery and control the condition number of the Fisher information matrix.
• Derives non-asymptotic bounds on the estimation error $\|\hat{\theta} - \theta^*\|_{\mathcal{F}^*}^2$ and $\ell_1$-error using the $\kappa_{\min}^*$ and $\kappa_{\max}^*$ parameters of the Fisher information matrix.
• Establishes that the final sparse estimate $\tilde{\theta}$ has support size at most $2s$ under appropriate thresholding and regularization.

Experimental results

Research questions

• RQ1 How fast must the sample size $n$ grow relative to the dimension $p$ for the prediction loss of a general exponential family to exhibit strong convexity?
• RQ2 Can $L_1$-regularized estimation in general exponential families achieve the same $O(s \log p / n)$ convergence rate as in linear regression?
• RQ3 What conditions on the sufficient statistics ensure that the empirical log-likelihood becomes strongly convex beyond a certain sample size?
• RQ4 Can a two-stage procedure recover a sparse model with $O(s)$ non-zero features while maintaining low prediction risk?
• RQ5 How does the condition number of the Fisher information matrix affect the sparsity and accuracy of $L_1$-regularized estimates in general exponential families?

Key findings

• The prediction loss of a general exponential family becomes strongly convex once the sample size exceeds a threshold that depends on the growth rate of standardized moments and cumulants, quantifying a 'burn-in' phase.
• Under the sub-Gaussian condition on sufficient statistics, the $L_1$-regularized estimator achieves a prediction loss bound of $O(s \log p / n)$ with high probability.
• The $\ell_1$-error of the $L_1$-regularized estimator is bounded by $O(\sigma s / \kappa_{\min}^{*2} \sqrt{\log p / n})$, where $\kappa_{\min}^*$ is the smallest eigenvalue of the Fisher information matrix.
• The two-stage procedure produces an estimate $\tilde{\theta}$ with at most $2s$ non-zero entries and a prediction risk bounded by $O((\kappa_{\max}^*/\kappa_{\min}^*)^2 \cdot s \sigma^2 \log p / n)$.
• The required sample size for reliable estimation scales as $n \geq K \alpha^{*2} \sigma^2 \log(p/\delta) \cdot \max\{ s \kappa_{\max}^{*2} / \kappa_{\min}^{*4}, \alpha^{*2} \|\theta^*\|_1^2 \}$, where $\alpha^*$ is the analytic standardized moment of $\theta^*$.
• The final sparse estimate $\tilde{\theta}$ achieves a risk bound that is within a constant factor of the optimal $O(s \log p / n)$ rate, up to a condition number penalty.