[论文解读] Regularization vs. Relaxation: A conic optimization perspective of statistical variable selection

Hongbo Dong, Kun Chen+1 more|arXiv (Cornell University)|Oct 20, 2015

Sparse and Compressive Sensing Techniques参考文献 46被引用 24

一句话总结

本文提出了一种锥优化框架，将稀疏变量选择视为混合整数二次规划（MIQP），表明MCP和反向Huber等流行非凸惩罚函数是视角松弛的特例。最紧致的松弛形式为半定规划（SDP），在凸性和ℓ₀-范数逼近之间实现最优平衡，Goemans-Williamson取整法提供了有效的近似解，其紧致性与解的质量均优于以往的凸松弛方法。

ABSTRACT

Variable selection is a fundamental task in statistical data analysis. Sparsity-inducing regularization methods are a popular class of methods that simultaneously perform variable selection and model estimation. The central problem is a quadratic optimization problem with an l0-norm penalty. Exactly enforcing the l0-norm penalty is computationally intractable for larger scale problems, so dif- ferent sparsity-inducing penalty functions that approximate the l0-norm have been introduced. In this paper, we show that viewing the problem from a convex relaxation perspective offers new insights. In particular, we show that a popular sparsity-inducing concave penalty function known as the Minimax Concave Penalty (MCP), and the reverse Huber penalty derived in a recent work by Pilanci, Wainwright and Ghaoui, can both be derived as special cases of a lifted convex relaxation called the perspective relaxation. The optimal perspective relaxation is a related minimax problem that balances the overall convexity and tightness of approximation to the l0 norm. We show it can be solved by a semidefinite relaxation. Moreover, a probabilistic interpretation of the semidefinite relaxation reveals connections with the boolean quadric polytope in combinatorial optimization. Finally by reformulating the l0-norm pe- nalized problem as a two-level problem, with the inner level being a Max-Cut problem, our proposed semidefinite relaxation can be realized by replacing the inner level problem with its semidefinite relaxation studied by Goemans and Williamson. This interpretation suggests using the Goemans-Williamson rounding procedure to find approximate solutions to the l0-norm penalized problem. Numerical experiments demonstrate the tightness of our proposed semidefinite relaxation, and the effectiveness of finding approximate solutions by Goemans-Williamson rounding.

研究动机与目标

• 将线性回归中的稀疏变量选择问题重新表述为带ℓ₀-范数惩罚的混合整数二次规划（MIQP）。
• 分析诱导稀疏性的惩罚函数（如MCP、反向Huber）与凸松弛之间的关系，特别是视角松弛的关系。
• 为MIQP形式化构建一个紧致的半定松弛，以最优方式平衡凸性与对ℓ₀-范数的逼近。
• 建立所提松弛与组合优化结构（如布尔二次多面体）之间的联系。
• 提出一种实用的求解策略：在稀疏回归问题的Max-Cut重表述基础上，对半定松弛应用Goemans-Williamson取整法。

提出的方法

• 将ℓ₀-惩罚回归问题建模为混合整数二次规划（MIQP），将变量选择视为离散优化问题。
• 引入视角松弛作为MIQP的凸松弛，并证明MCP与反向Huber惩罚均为其特例。
• 将最优视角松弛形式化为一个极小极大问题，以在对ℓ₀-范数的紧致逼近与凸性之间实现平衡，该问题可通过半定规划（SDP）求解。
• 将稀疏回归问题重述为两级优化问题，其中内层对应于Max-Cut问题。
• 用Goemans与Williamson提出的Max-Cut问题的半定松弛替代内层Max-Cut问题，从而获得可计算的外层松弛。
• 对SDP解应用Goemans-Williamson取整过程，生成原始ℓ₀-惩罚问题的近似解。

实验结果

研究问题

• RQ1 广泛使用的非凸惩罚函数MCP与反向Huber能否被解释为统一凸松弛框架下的特例？
• RQ2 ℓ₀-惩罚回归问题的最紧致凸松弛是什么？它能否被形式化为一个极小极大问题？
• RQ3 所提出的半定松弛在紧致性上与现有凸松弛（如反向Huber松弛）相比如何？
• RQ4 Goemans-Williamson取整过程能否有效应用于通过半定松弛求解稀疏回归问题的近似解？
• RQ5 求解所提出的SDP松弛的计算可扩展性如何？其随预测变量数量p的增加如何变化？

主要发现

• 数值实验表明，所提出的半定松弛显著优于基于反向Huber惩罚的凸松弛，紧致性更高。
• Goemans-Williamson取整生成的最优目标值上界，平均仅比最佳已知上界（τ_UB）低0.34%，且在λ和μ值较高时，相对差异降至0.00%。
• 当p = 800时，使用DSDP求解SDP的平均计算时间约为279秒，表明在大规模问题上存在可扩展性挑战。
• 该半定松弛可从概率角度解释为对重缩放多元伯努利变量的矩匹配问题，与组合优化中的布尔二次多面体相关联。
• 该松弛在极小极大意义下实现了凸性与对ℓ₀-范数逼近质量之间的平衡，为启发式惩罚函数提供了一个有原则的替代方案。
• Goemans-Williamson取整过程始终能产生高质量解，在测试实例上，Gurobi在60秒内无法进一步改进结果，表明其作为启发式方法的有效性。