[Paper Review] A Tutorial on Libra: R package for the Linearized Bregman Algorithm in High Dimensional Statistics

Jiechao Xiong, Feng Ruan+1 more|arXiv (Cornell University)|Apr 20, 2016

Statistical Methods and Inference13 references3 citations

TL;DR

This paper introduces the Libra R package, implementing the Linearized Bregman Algorithm for high-dimensional sparse statistical modeling. It enables efficient computation of sparse regularization paths in linear and logistic regression, Gaussian, Ising, and Potts graphical models, offering a bias-free alternative to LASSO through early-stopped iterative thresholding with theoretical consistency guarantees under similar conditions.

ABSTRACT

The R package, Libra, stands for the LInearized BRegman Al- gorithm in high dimensional statistics. The Linearized Bregman Algorithm is a simple iterative procedure to generate sparse regularization paths of model estimation, which are rstly discovered in applied mathematics for image restoration and particularly suitable for parallel implementation in large scale problems. The limit of such an algorithm is a sparsity-restricted gradient descent ow, called the Inverse Scale Space, evolving along a par- simonious path of sparse models from the null model to over tting ones. In sparse linear regression, the dynamics with early stopping regularization can provably meet the unbiased Oracle estimator under nearly the same condition as LASSO, while the latter is biased. Despite their successful applications, statistical consistency theory of such dynamical algorithms remains largely open except for some recent progress on linear regression. In this tutorial, algorithmic implementations in the package are discussed for several widely used sparse models in statistics, including linear regression, logistic regres- sion, and several graphical models (Gaussian, Ising, and Potts). Besides the simulation examples, various application cases are demonstrated, with real world datasets from diabetes, publications of COPSS award winners, as well as social networks of two Chinese classic novels, Journey to the West and Dream of the Red Chamber.

Motivation & Objective

• To provide a practical, scalable implementation of the Linearized Bregman Algorithm for high-dimensional statistical modeling.
• To address the bias issue in traditional penalized estimators like LASSO by offering an alternative that achieves oracle properties under similar conditions.
• To support a wide range of sparse models, including linear and logistic regression, and Gaussian, Ising, and Potts graphical models.
• To facilitate model selection via regularization paths using iterative thresholding with early stopping.
• To bridge theoretical advances in inverse scale space dynamics with practical statistical applications through an accessible R package.

Proposed method

• The algorithm uses iterative updates based on the proximal operator of a penalty function, with dynamics defined by $\theta^{k+1} = \kappa \cdot \text{prox}_P(z^{k+1})$ and $z^{k+1} = z^k - \alpha_k \nabla_\theta L(\theta^k)$.
• It generates a sparse regularization path starting from the null model, evolving toward overfitting models through iterative shrinkage and gradient descent.
• The method is grounded in the Inverse Scale Space (ISS) limit, where $\frac{d\rho}{dt} = -\nabla_\theta L(\theta(t))$ and $\rho(t) \in \partial P(\theta(t))$, ensuring convergence to an unbiased oracle estimator.
• The package supports various loss functions: least squares for linear regression, log-likelihood for logistic and multinomial models, and Potts model likelihood for categorical data.
• For non-grid time points in the path, linear interpolation is applied to $z(t)$ and $\theta(t)$ to enable evaluation at user-specified $t$ values in `tlist`.
• The algorithm initializes the intercept $\theta_0^0$ as the minimizer of the loss at $\theta=0$, avoiding spurious variable selection due to intercept effects.

Experimental results

Research questions

• RQ1 Can the Linearized Bregman Algorithm produce sparse, consistent estimators in high-dimensional linear models with reduced bias compared to LASSO?
• RQ2 How does the algorithm’s regularization path compare to traditional penalized M-estimation in terms of model selection accuracy and computational efficiency?
• RQ3 What is the statistical consistency of the early-stopped Linearized Bregman iteration in non-linear models such as logistic and multinomial regression?
• RQ4 Can the algorithm be effectively extended to graphical models, including Ising and Potts models, for sparse structure learning?
• RQ5 How does the choice of initialization, particularly for the intercept, affect variable selection stability in unbalanced data settings?

Key findings

• The Linearized Bregman Algorithm with early stopping achieves model selection consistency and produces an unbiased oracle estimator under conditions nearly identical to those required for LASSO, despite LASSO's inherent bias.
• The algorithm generates a parsimonious path of sparse models from the null model to overfitting ones, enabling effective regularization path analysis.
• For logistic regression, the package supports both binomial and multinomial models, with gradient computation costing $O(np^2K^2)$ for $K$-class outcomes.
• In Potts model estimation, the package supports group sparsity penalties via $\sum_{k=1}^p \sqrt{\sum_{s,t} \theta_{js,kt}^2}$, enabling structured sparsity in categorical data.
• The default path is a geometric sequence of $nt$ time points from $t_0$ to $t_0 \cdot \text{trate}$, ensuring dense sampling of the regularization path.
• Linear interpolation is used to compute $\theta(t)$ at non-grid time points in `tlist`, enabling flexible evaluation of the regularization path.