[Paper Review] Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

Zhouchen Lin, Risheng Liu+1 more|arXiv (Cornell University)|Sep 2, 2011

Sparse and Compressive Sensing Techniques16 references753 citations

TL;DR

This paper proposes LADMAP, a linearized alternating direction method with adaptive penalty for solving low-rank representation (LRR) efficiently. By linearizing the quadratic penalty and using adaptive penalty updates, LADMAP reduces the computational complexity from O(n³) to O(rn²), enabling scalable large-scale LRR applications with faster convergence than state-of-the-art methods.

ABSTRACT

Many machine learning and signal processing problems can be formulated as lin-early constrained convex programs, which could be efficiently solved by the alter-nating direction method (ADM). However, usually the subproblems in ADM are easily solvable only when the linear mappings in the constraints are identities. To address this issue, we propose a linearized ADM (LADM) method by linearizing the quadratic penalty term and adding a proximal term when solving the sub-problems. For fast convergence, we also allow the penalty to change adaptively according a novel update rule. We prove the global convergence of LADM with adaptive penalty (LADMAP). As an example, we apply LADMAP to solve low-rank representation (LRR), which is an important subspace clustering technique yet suffers from high computation cost. By combining LADMAP with a skinny SVD representation technique, we are able to reduce the complexity O(n3) of the original ADM based method to O(rn2), where r and n are the rank and size of the representation matrix, respectively, hence making LRR possible for large scale applications. Numerical experiments verify that for LRR our LADMAP based methods are much faster than state-of-the-art algorithms. 1

Motivation & Objective

• To address the high computational cost of existing alternating direction method (ADM) approaches for low-rank representation (LRR).
• To overcome the limitation of ADM subproblems being solvable only when linear mappings are identities.
• To develop a linearized ADM variant with adaptive penalty for faster convergence and improved efficiency.
• To reduce the computational complexity of LRR from O(n³) to O(rn²) for large-scale applications.
• To enable practical deployment of LRR in large-scale machine learning and signal processing tasks.

Proposed method

• Proposes a linearized alternating direction method (LADM) by linearizing the quadratic penalty term in the augmented Lagrangian function.
• Introduces a proximal term in subproblems to improve numerical stability and convergence.
• Employs an adaptive penalty update rule that dynamically adjusts the penalty parameter based on convergence progress.
• Combines LADMAP with a skinny SVD representation technique to reduce computational cost.
• Solves the subproblems efficiently by exploiting the structure of low-rank matrix optimization.
• Proves global convergence of the proposed LADMAP algorithm under standard assumptions.

Experimental results

Research questions

• RQ1 Can a linearized ADM with adaptive penalty achieve faster convergence for low-rank representation compared to traditional ADM methods?
• RQ2 How can the computational complexity of LRR be reduced from O(n³) to O(rn²) while preserving solution accuracy?
• RQ3 What is the impact of adaptive penalty updating on the convergence speed and robustness of the algorithm?
• RQ4 Can the proposed method scale effectively to large-scale datasets where standard ADM fails due to high cost?
• RQ5 How does the combination of linearization and skinny SVD improve the efficiency of low-rank representation?

Key findings

• LADMAP achieves global convergence under standard assumptions, ensuring reliable solution quality.
• The computational complexity of LRR is reduced from O(n³) to O(rn²), where r is the rank and n is the matrix size.
• The proposed method significantly outperforms state-of-the-art algorithms in terms of runtime on large-scale datasets.
• Numerical experiments confirm that LADMAP converges faster than existing ADM-based methods for LRR.
• The adaptive penalty update rule enhances convergence speed without compromising solution accuracy.