[Paper Review] Relaxed Majorization-Minimization for Non-smooth and Non-convex Optimization

Chen Xu, Zhouchen Lin+2 more|arXiv (Cornell University)|Nov 25, 2015

Sparse and Compressive Sensing Techniques24 references3 citations

TL;DR

This paper proposes a relaxed majorization-minimization (MM) framework for non-smooth and non-convex optimization that only requires the directional derivative difference between the objective and surrogate function to vanish asymptotically. The method enables direct approximation of the full non-smooth objective, outperforms state-of-the-art methods in robust matrix factorization (RMF) with guaranteed convergence to stationary points, and achieves superior solution quality and faster convergence.

ABSTRACT

We propose a new majorization-minimization (MM) method for non-smooth and non-convex programs, which is general enough to include the existing MM methods. Besides the local majorization condition, we only require that the difference between the directional derivatives of the objective function and its surrogate function vanishes when the number of iterations approaches infinity, which is a very weak condition. So our method can use a surrogate function that directly approximates the non-smooth objective function. In comparison, all the existing MM methods construct the surrogate function by approximating the smooth component of the objective function. We apply our relaxed MM methods to the robust matrix factorization (RMF) problem with different regularizations, where our locally majorant algorithm shows advantages over the state-of-the-art approaches for RMF. This is the first algorithm for RMF ensuring, without extra assumptions, that any limit point of the iterates is a stationary point.

Motivation & Objective

• To address the limitations of existing majorization-minimization (MM) methods that require strong surrogate conditions and only approximate smooth components.
• To develop a general MM framework that can directly approximate non-smooth and non-convex objective functions without splitting into smooth and non-smooth parts.
• To ensure convergence of iterates to stationary points for robust matrix factorization (RMF) without additional assumptions.
• To improve solution quality and convergence speed in RMF with various regularizations compared to state-of-the-art methods.

Proposed method

• Proposes a relaxed condition on the surrogate function: the directional derivative difference between the objective and surrogate must vanish as the number of iterations approaches infinity.
• Introduces a locally majorant surrogate function that directly approximates the full non-smooth objective function, unlike prior methods that only approximate the smooth part.
• Applies the relaxed MM framework to robust matrix factorization (RMF) with ℓ1 and nuclear norm regularization, using iterative minimization of the surrogate.
• Employs a convergence analysis proving that any limit point of the iterates is a stationary point under the relaxed condition.
• Uses a block-coordinate descent strategy in RMF to solve the surrogate subproblems efficiently.
• Employs initialization via PCA and k-means to improve convergence and solution quality in NMF experiments.

Experimental results

Research questions

• RQ1 Can a majorization-minimization method be designed to directly approximate non-smooth and non-convex objectives without requiring smoothness of the surrogate difference?
• RQ2 Does the proposed relaxed condition on the directional derivative difference ensure convergence to stationary points in non-convex, non-smooth settings?
• RQ3 How does the relaxed MM method compare to state-of-the-art algorithms in terms of solution quality and convergence speed for robust matrix factorization?
• RQ4 Can the proposed method guarantee convergence to stationary points in RMF without additional assumptions?
• RQ5 What is the performance gain of the relaxed MM method in real-world applications like motion tracking and face clustering?

Key findings

• The proposed relaxed MM method guarantees that any limit point of the iterates is a stationary point for robust matrix factorization, which is the first such guarantee without extra assumptions.
• In motion tracking, RMF-LMMM achieved the lowest relative error (9.69) and produced visually closed, circular tracks, outperforming UNuBi and RMF-GMMM.
• In face clustering, RMF-LMMM achieved the highest accuracy (0.7250), NMI (0.8655), and purity (0.7500) on the AT&T dataset, significantly outperforming ℓ1-NMF and RMF-GMMM.
• On the CMUPIE dataset, RMF-LMMM achieved a relative error of 4.67, the lowest among all methods, and a purity of 0.4456, the highest in the benchmark.
• The method converged in only 20 iterations on average, significantly faster than ℓ1-NMF (5000 iterations) and RMF-GMMM (500 iterations), demonstrating superior convergence speed.
• The relaxed MM framework generalizes existing MM methods and allows direct approximation of the full non-smooth objective, enabling better performance than prior surrogate construction strategies.