[Paper Review] Iterative Reweighted Singular Value Minimization Methods for $l_p$ Regularized Unconstrained Matrix Minimization

Zhaosong Lu, Yong Zhang|arXiv (Cornell University)|Jan 5, 2014

Advanced Optimization Algorithms Research3 citations

TL;DR

This paper proposes iterative reweighted singular value minimization (IRSVM) and nonmonotone proximal gradient (NPG) methods for solving $l_p$ regularized unconstrained matrix minimization problems. It establishes that all local minimizers are first-order stationary points, derives lower bounds on nonzero singular values of these points, and proves global convergence of both algorithms, with IRSVM showing superior speed and solution quality in experiments.

ABSTRACT

In this paper we study general lp regularized unconstrained matrix minimization problems. In particular, we first introduce a class of first-order stationary points for them. And we show that the first-order stationary points introduced in [11] for an lp regularized vector minimization problem are equivalent to those of an lp regularized ma-trix minimization reformulation. We also establish that any local minimizer of the lp regularized matrix minimization problems must be a first-order stationary point. More-over, we derive lower bounds for nonzero singular values of the first-order stationary points and hence also of the local minimizers for the lp matrix minimization problems. The iterative reweighted singular value minimization (IRSVM) approaches are then pro-posed to solve these problems in which each subproblem has a closed-form solution. We show that any accumulation point of the sequence generated by these methods is a first-order stationary point of the problems. In addition, we study a nonmontone proximal gradient (NPG) method for solving the lp matrix minimization problems and establish its global convergence. Our computational results demonstrate that the IRSVM and NPG methods generally outperform some existing state-of-the-art methods in terms of solution quality and/or speed. Moreover, the IRSVM methods are slightly faster than the NPG method. Key words: lp regularized matrix minimization, iterative reweighted singular value min-imization, iterative reweighted least squares, nonmonotone proximal gradient method

Motivation & Objective

• To analyze first-order stationary points in $l_p$ regularized unconstrained matrix minimization problems.
• To establish that local minimizers of the $l_p$ matrix minimization problem must be first-order stationary points.
• To derive lower bounds on the nonzero singular values of first-order stationary points and local minimizers.
• To develop iterative reweighted singular value minimization (IRSVM) methods with closed-form subproblems.
• To propose and analyze a nonmonotone proximal gradient (NPG) method for the same class of problems with global convergence guarantees.

Proposed method

• Introduces a class of first-order stationary points for $l_p$ regularized matrix minimization, linking them to equivalent points in vector-based formulations.
• Derives theoretical lower bounds on the nonzero singular values of first-order stationary points and local minimizers.
• Proposes IRSVM methods where each subproblem is solved in closed form via iterative reweighting of singular values.
• Employs a nonmonotone line search strategy in the NPG method to enhance convergence robustness.
• Uses a proximal term in the NPG subproblems to stabilize iterates and ensure convergence.
• Proves that any accumulation point of the IRSVM and NPG iterates is a first-order stationary point of the original problem.

Experimental results

Research questions

• RQ1 What characterizes the first-order stationary points in $l_p$ regularized unconstrained matrix minimization problems?
• RQ2 How do the singular values of first-order stationary points and local minimizers behave, and can lower bounds be derived?
• RQ3 Can iterative reweighted singular value minimization (IRSVM) be designed with closed-form solutions and global convergence?
• RQ4 Does a nonmonotone proximal gradient (NPG) method achieve global convergence for $l_p$ matrix minimization?
• RQ5 How do IRSVM and NPG compare in solution quality and computational speed to existing state-of-the-art methods?

Key findings

• All local minimizers of the $l_p$ regularized matrix minimization problem are first-order stationary points.
• Nonzero singular values of first-order stationary points and local minimizers are bounded below by a positive value depending on the problem parameters.
• The IRSVM method generates iterates whose accumulation points are first-order stationary points, with each subproblem solvable in closed form.
• The NPG method is globally convergent to a first-order stationary point, even without requiring monotonic decrease in the objective.
• Computational results show that IRSVM and NPG outperform existing state-of-the-art methods in both solution quality and speed.
• IRSVM methods are slightly faster than the NPG method while maintaining comparable or better solution accuracy.