[Paper Review] Submodular meets Spectral: Greedy Algorithms for Subset Selection, Sparse Approximation and Dictionary Selection

Abhimanyu Das, David Kempe|arXiv (Cornell University)|Feb 19, 2011

Sparse and Compressive Sensing Techniques15 references270 citations

TL;DR

This paper introduces the submodularity ratio as a new measure to explain why greedy algorithms perform well in subset selection, sparse approximation, and dictionary selection—even with highly correlated or near-singular data. By combining submodular function analysis with spectral techniques, the authors provide the strongest known approximation guarantees, showing that the submodularity ratio is a superior predictor of greedy algorithm performance compared to traditional spectral parameters like coherence or eigenvalues.

ABSTRACT

We study the problem of selecting a subset of k random variables from a large set, in order to obtain the best linear prediction of another variable of interest. This problem can be viewed in the context of both feature selection and sparse approximation. We analyze the performance of widely used greedy heuristics, using insights from the maximization of submodular functions and spectral analysis. We introduce the submodularity ratio as a key quantity to help understand why greedy algorithms perform well even when the variables are highly correlated. Using our techniques, we obtain the strongest known approximation guarantees for this problem, both in terms of the submodularity ratio and the smallest k-sparse eigenvalue of the covariance matrix. We further demonstrate the wide applicability of our techniques by analyzing greedy algorithms for the dictionary selection problem, and significantly improve the previously known guarantees. Our theoretical analysis is complemented by experiments on real-world and synthetic data sets; the experiments show that the submodularity ratio is a stronger predictor of the performance of greedy algorithms than other spectral parameters.

Motivation & Objective

• To explain the strong empirical performance of greedy algorithms in subset selection and sparse approximation, especially when data are highly correlated or near-singular.
• To develop a theoretical framework that bridges the gap between worst-case approximation bounds and practical performance of greedy algorithms.
• To improve approximation guarantees for greedy algorithms in subset selection and dictionary selection using a novel measure of approximate submodularity.
• To demonstrate that the submodularity ratio is a stronger predictor of greedy algorithm performance than traditional spectral parameters such as coherence, condition number, or restricted isometry property (RIP).
• To refine theoretical analysis by focusing only on relevant sets during greedy iterations, leading to significantly improved bounds in practice.

Proposed method

• Introduce the submodularity ratio as a measure of how close the R² objective function is to being submodular, quantifying deviation from submodularity on a given input.
• Combine submodular function analysis with spectral techniques, particularly leveraging the smallest k-sparse eigenvalue of the covariance matrix to derive approximation guarantees.
• Formalize the performance of Forward Regression and Orthogonal Matching Pursuit (OMP) using the submodularity ratio, proving that greedy algorithms achieve a (1 - e^(-γ)) approximation ratio, where γ is the submodularity ratio.
• Introduce a refined analysis that prunes sets with low marginal contribution (≤ ε) during submodularity ratio computation, leading to significantly improved empirical bounds.
• Use real-world and synthetic datasets to empirically validate that the submodularity ratio correlates strongly with greedy algorithm performance, outperforming traditional spectral metrics.
• Apply the framework to the dictionary selection problem, significantly improving existing theoretical guarantees for greedy algorithms in this context.

Experimental results

Research questions

• RQ1 Why do greedy algorithms like Forward Regression and OMP perform well in practice despite being NP-hard problems with weak worst-case bounds?
• RQ2 To what extent does the performance of greedy algorithms depend on the spectral properties of the covariance matrix, such as coherence or condition number?
• RQ3 Can a new measure of approximate submodularity explain the robustness of greedy algorithms in the presence of highly correlated or near-singular data?
• RQ4 How does the submodularity ratio compare to other spectral parameters in predicting the performance of greedy algorithms?
• RQ5 Can a refined analysis focusing only on high-contribution sets improve theoretical bounds and close the gap between theory and practice?

Key findings

• The submodularity ratio is a significantly stronger predictor of greedy algorithm performance than coherence, condition number, or RIP-based bounds.
• For real-world datasets, the submodularity ratio remains high (e.g., >0.8) even when theoretical bounds are weak, explaining near-optimal performance of Forward Regression.
• After pruning low-marginal-contribution sets, the refined submodularity ratio estimate exceeds 0.8 for most real-world cases, reducing the gap between theory and practice.
• Theoretical guarantees based on the submodularity ratio are stronger than previous results, especially in high-correlation or near-singular settings where traditional spectral bounds fail.
• The submodularity ratio explains why greedy algorithms avoid strong supermodular behavior that could hurt performance, as evidenced by concave R² plots in synthetic data.
• The framework extends to dictionary selection, providing significantly improved approximation guarantees for greedy algorithms in this problem class.