[Paper Review] Two New Algorithms for Solving Covariance Graphical Lasso Based on Coordinate Descent and ECM

Hao Wang|arXiv (Cornell University)|May 18, 2012

Statistical Methods and Inference3 references3 citations

TL;DR

This paper proposes two new algorithms—coordinate descent and ECM—for solving the covariance graphical lasso problem, which estimates sparse covariance matrices using an L1 penalty to identify marginal independence structures. The proposed methods are simpler, faster, and more stable than the existing algorithm by Bien & Tibshirani (2011), with empirical results showing superior performance in speed, convergence, and sparsity recovery across various simulation settings.

ABSTRACT

Covariance graphical lasso applies a lasso penalty on the elements of the covariance matrix. This method is useful because it not only produces sparse estimation of covariance matrix but also discovers marginal independence structures by generating zeros in the covariance matrix. We propose and explore two new algorithms for solving the covariance graphical lasso problem. Our new algorithms are based on coordinate descent and ECM. We show that these two algorithms are more attractive than the only existing competing algorithm of Bien and Tibshirani (2011) in terms of simplicity, speed and stability. We also discuss convergence properties of our algorithms.

Motivation & Objective

• To address the computational challenges of minimizing the non-convex objective function in covariance graphical lasso, which involves an L1 penalty on the covariance matrix.
• To develop simpler, faster, and more numerically stable algorithms than the existing majorize-minimize approach by Bien & Tibshirani (2011).
• To enable efficient estimation of sparse covariance matrices that reveal marginal independence structures through zero off-diagonal elements.
• To extend the applicability of coordinate descent and ECM algorithms—previously used in other regularized models—to the non-convex covariance graphical lasso problem.
• To provide open-source implementations in MATLAB and R for broader adoption and reproducibility.

Proposed method

• The coordinate descent algorithm updates one column and row of the covariance matrix at a time, fixing all other elements, using block matrix inversion to simplify the objective function.
• The ECM algorithm iteratively performs expectation and conditional maximization steps to optimize the objective function, leveraging the structure of the covariance matrix.
• Both algorithms are applied to the non-convex objective: minimize log(det(Σ)) + tr(SΣ⁻¹) + ρ||Σ||₁ over positive definite matrices, with Σ the covariance matrix and S the sample covariance matrix.
• The coordinate descent method uses a block-wise optimization strategy based on partitioning Σ into submatrices and updating β and γ, the last column/row and its variance.
• The ECM algorithm alternates between estimating the conditional expectation and maximizing the expected log-likelihood under a majorizing function.
• Convergence is established for both algorithms, with the coordinate descent method shown to monotonically decrease the objective function at each step.

Experimental results

Research questions

• RQ1 Can coordinate descent and ECM be effectively adapted to solve the non-convex covariance graphical lasso problem with an L1 penalty?
• RQ2 How do the proposed algorithms compare in speed, stability, and convergence to the majorize-minimize algorithm of Bien & Tibshirani (2011)?
• RQ3 Does the choice of initial value (full vs. diagonal) significantly affect the performance and sparsity of the estimated covariance matrix?
• RQ4 Can the new algorithms achieve lower objective function values and better sparsity patterns than the existing method?
• RQ5 What is the computational efficiency of the proposed algorithms across different sample sizes and sparsity levels?

Key findings

• The coordinate descent algorithm consistently achieved the lowest objective function values across all simulation settings, including cases where the L1 penalty was small and sparsity was low.
• The ECM algorithm was significantly faster than Bien & Tibshirani’s (2011) algorithm, with median runtimes 2–5 times lower in most scenarios.
• The coordinate descent method required more CPU time than ECM in high-dimensional settings (p=200), but still outperformed the benchmark in objective function minimization and stability.
• Initial values had a strong impact: diagonal starting matrices produced sparser solutions than full initial matrices, reflecting the influence of initialization on sparsity.
• Bien & Tibshirani’s (2011) algorithm was highly sensitive to initialization and required careful tuning, often failing to converge or producing poor results with default settings.
• The proposed algorithms were numerically more stable than the benchmark, with no instances of divergence or failure to converge in the tested simulations.