[논문 리뷰] Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

Tian Tong, Cong Ma외 1명|arXiv (Cornell University)|2020. 05. 18.

Sparse and Compressive Sensing Techniques인용 수 39

한 줄 요약

Scaled Gradient Descent(ScaledGD)를 소개합니다. 이는 선행조건화된 분해 형태의 그래디언트 방법으로, 조건수와 무관하게 선형 수렴을 달성하며 악조건의 저랭크 행렬 추정 문제에서 각 반복의 비용이 가볍습니다.

ABSTRACT

Low-rank matrix estimation is a canonical problem that finds numerous applications in signal processing, machine learning and imaging science. A popular approach in practice is to factorize the matrix into two compact low-rank factors, and then optimize these factors directly via simple iterative methods such as gradient descent and alternating minimization. Despite nonconvexity, recent literatures have shown that these simple heuristics in fact achieve linear convergence when initialized properly for a growing number of problems of interest. However, upon closer examination, existing approaches can still be computationally expensive especially for ill-conditioned matrices: the convergence rate of gradient descent depends linearly on the condition number of the low-rank matrix, while the per-iteration cost of alternating minimization is often prohibitive for large matrices. The goal of this paper is to set forth a competitive algorithmic approach dubbed Scaled Gradient Descent (ScaledGD) which can be viewed as pre-conditioned or diagonally-scaled gradient descent, where the pre-conditioners are adaptive and iteration-varying with a minimal computational overhead. With tailored variants for low-rank matrix sensing, robust principal component analysis and matrix completion, we theoretically show that ScaledGD achieves the best of both worlds: it converges linearly at a rate independent of the condition number of the low-rank matrix similar as alternating minimization, while maintaining the low per-iteration cost of gradient descent. Our analysis is also applicable to general loss functions that are restricted strongly convex and smooth over low-rank matrices. To the best of our knowledge, ScaledGD is the first algorithm that provably has such properties over a wide range of low-rank matrix estimation tasks.

연구 동기 및 목표

• 저차원 행렬 추정에서 악조건 하에서의 더 빠르고 조건수에 독립적인 수렴의 필요성을 동기화합니다.
• 저지표 오버헤드로 대각선 스케일링된 적응형 전처리된 그래디언트 방법으로 ScaledGD를 제안합니다.
• 주요 문제들(행렬 sensing, robust PCA, 행렬 완성)에서 조건수와 무관한 선형 수렴의 이론적 보장을 보입니다.
• 전처리기를 고려한 새로운 거리 메트릭을 통해 통일된 분석을 제공합니다.
• 다수의 작업에서 일반 그래디언트 디센트보다 Empirical 가속을 입증합니다.]
• method
• ["X를 LR^T로 매개변수화하여 랭크-r 분해를 얻고 L과 R을 함께 최적화합니다.","그래디언트 스텝에 대각 스케일링을 적용하고, (R^T R)^{-1}과 (L^T L)^{-1}을 사용하여 스케일링된 업데이트를 형성합니다.","전처리기와 GL(r) 동작에 의한 동일성까지 고려하는 새로운 거리 메트릭을 도입합니다.","스케일드GD를 세 가지 작업: 행렬 sensing, robust PCA, 행렬 완성에 대해 스펙트럼 초기화와 함께 구현합니다.","새로운 거리 메트릭 하에서 ScaledGD 반복의 수축성(contracting)을 보이고, 조건수에 독립적인 선형 수렴 속도를 얻습니다.","일반적 손실 함수에 대해서도 저랭크 행렬 위에서 제한적으로 강볼록하고 매끄러운 경우에 대한 분석을 일반화합니다."]
• research_questions1[
• Can a first-order method with per-iteration costs similar to gradient descent achieve linear convergence that is independent of the condition number for ill-conditioned low-rank matrix estimation?
• What preconditioning strategy yields fast, provable convergence across matrix sensing, robust PCA, and matrix completion?
• How does the proposed distance metric capture the effect of preconditioning on convergence?
• Does scalability hold under standard sampling/models (RIP, incoherence, etc.) while maintaining near-optimal sample complexity?

제안 방법

• Parametrize X as LR^T to obtain a rank-r factorization and optimize over L and R jointly.
• Apply a diagonal scaling to gradient steps, using (R^T R)^{-1} and (L^T L)^{-1} to form the scaled updates.
• Introduce a new distance metric that accounts for the preconditioners and the identifiability up to GL(r) actions.
• Instantiate ScaledGD for three tasks: matrix sensing, robust PCA, and matrix completion, with spectral initialization.
• Prove contraction of the ScaledGD iterates under the new distance metric, yielding linear convergence with rate independent of the condition number.
• Generalize the analysis to general loss functions that are restricted strongly convex and smooth over low-rank matrices.

실험 결과

연구 질문

• RQ1 Can a first-order method with per-iteration costs similar to gradient descent achieve linear convergence that is independent of the condition number for ill-conditioned low-rank matrix estimation?
• RQ2 What preconditioning strategy yields fast, provable convergence across matrix sensing, robust PCA, and matrix completion?
• RQ3 How does the proposed distance metric capture the effect of preconditioning on convergence?
• RQ4 Does scalability hold under standard sampling/models (RIP, incoherence, etc.) while maintaining near-optimal sample complexity?

주요 결과

• ScaledGD achieves linear convergence with a rate that does not depend on the condition number κ, under suitable initialization.
• For matrix sensing, ScaledGD converges in O(log(1/ε)) iterations with m = O(n r^2 κ^2) measurements and RIP with δ_{2r} ≤ 0.02/(√r κ).
• For robust PCA, ScaledGD with hard thresholding converges in O(log(1/ε)) iterations under α ≤ c/(μ r^{3/2} κ) corruption, assuming μ-incoherence.
• For matrix completion, ScaledGD attains ε-accuracy in O(log(1/ε)) iterations with sample complexity n1 n2 p ≳ (μ κ^2 ∨ log n) μ n r^2 κ^2.
• ScaledGD does not require regularizations to balance L and R or extra incoherence projections, unlike some prior methods.
• The framework extends to general loss functions that are restricted strongly convex and smooth over low-rank matrices.