[Paper Review] Towards Optimal Sparse Inverse Covariance Selection through Non-Convex Optimization.

Sidhant Misra, Marc Vuffray+2 more|arXiv (Cornell University)|Mar 15, 2017

Statistical Methods and Inference1 citations

TL;DR

This paper proposes DICE and SLICE, two algorithms for sparse inverse covariance selection that achieve sample complexity matching the information-theoretic lower bound up to a constant factor. DICE achieves optimal sample complexity by solving a non-convex optimization problem, while SLICE offers a practical mixed-integer quadratic program formulation with comparable theoretical guarantees, both depending only on parameters in the lower bound: p (nodes), d (max degree), and κ (minimum edge strength).

ABSTRACT

What is the optimal number of independent observations from which a sparse Gaussian Graphical Model can be correctly recovered? Information-theoretic arguments provide a lower bound on the minimum number of samples necessary to perfectly identify the support of any multivariate normal distribution as a function of model parameters. For a model defined on a sparse graph with $p$ nodes, a maximum degree $d$ and minimum normalized edge strength $\kappa$, this necessary number of samples scales at least as $d \log p/\kappa^2$. The sample complexity requirements of existing methods for perfect graph reconstruction exhibit dependency on additional parameters that do not enter in the lower bound. The question of whether the lower bound is tight and achievable by a polynomial time algorithm remains open. In this paper, we constructively answer this question and propose an algorithm, termed DICE, whose sample complexity matches the information-theoretic lower bound up to a universal constant factor. We also propose a related algorithm SLICE that has a slightly higher sample complexity, but can be implemented as a mixed integer quadratic program which makes it attractive in practice. Importantly, SLICE retains a critical advantage of DICE in that its sample complexity only depends on quantities present in the information theoretic lower bound. We anticipate that this result will stimulate future search of computationally efficient sample-optimal algorithms.

Motivation & Objective

• To close the gap between the information-theoretic lower bound on sample complexity and existing algorithms for sparse inverse covariance selection.
• To develop a polynomial-time algorithm that matches the lower bound of d log p / κ² up to a universal constant.
• To ensure the algorithm's sample complexity depends only on parameters present in the information-theoretic lower bound: p, d, and κ.
• To provide a practical variant, SLICE, that maintains theoretical optimality while being implementable as a mixed-integer quadratic program.
• To stimulate future development of computationally efficient, sample-optimal algorithms for high-dimensional graphical model recovery.

Proposed method

• DICE formulates sparse inverse covariance selection as a non-convex optimization problem designed to recover the true graph support under minimal sample requirements.
• The algorithm leverages a non-convex penalty function that encourages sparsity while preserving edge strength information.
• DICE's optimization framework is constructed to align with the information-theoretic lower bound, ensuring sample complexity is tight up to a constant.
• SLICE is derived as a relaxation of DICE, reformulating the problem into a mixed-integer quadratic program for practical implementability.
• Both algorithms are designed so that their sample complexity depends only on p (number of variables), d (maximum node degree), and κ (minimum normalized edge strength), matching the lower bound parameters.
• Theoretical analysis proves that DICE achieves sample complexity within a universal constant factor of the information-theoretic lower bound.

Experimental results

Research questions

• RQ1 Is the information-theoretic lower bound of d log p / κ² on sample complexity for sparse Gaussian graphical model recovery tight and achievable by a polynomial-time algorithm?
• RQ2 Can a non-convex optimization approach be designed to achieve this optimal sample complexity without relying on additional parameters?
• RQ3 Does a practical algorithm exist that maintains sample-optimality while being implementable via mixed-integer programming?
• RQ4 Can the sample complexity depend only on the parameters present in the information-theoretic lower bound: p, d, and κ?
• RQ5 What is the trade-off between theoretical optimality and computational feasibility in sparse inverse covariance selection?

Key findings

• DICE achieves sample complexity matching the information-theoretic lower bound of d log p / κ² up to a universal constant factor.
• The sample complexity of DICE depends only on p, d, and κ—parameters that also define the lower bound—making it theoretically optimal.
• SLICE, while having slightly higher sample complexity, is implementable as a mixed-integer quadratic program, offering practical utility.
• Both DICE and SLICE retain the critical property that their sample complexity depends only on the parameters in the information-theoretic lower bound.
• The results demonstrate that sample-optimality in sparse inverse covariance selection is achievable with polynomial-time algorithms.
• The work provides a constructive answer to the open question of whether the lower bound is tight and attainable via efficient computation.