[Paper Review] Estimation of low-rank tensors via convex optimization

Ryota Tomioka, Kohei Hayashi+1 more|arXiv (Cornell University)|Oct 5, 2010

Sparse and Compressive Sensing Techniques37 references159 citations

TL;DR

This paper proposes three convex optimization approaches for estimating low-rank tensors via trace norm regularization, enabling automatic rank estimation and global convergence. The methods outperform conventional EM-based Tucker decomposition in predictive accuracy, speed, and recovery of multilinear structure on synthetic and real-world datasets, with one approach showing a sharp phase transition in reconstruction performance at low sampling rates.

ABSTRACT

In this paper, we propose three approaches for the estimation of the Tucker decomposition of multi-way arrays (tensors) from partial observations. All approaches are formulated as convex minimization problems. Therefore, the minimum is guaranteed to be unique. The proposed approaches can automatically estimate the number of factors (rank) through the optimization. Thus, there is no need to specify the rank beforehand. The key technique we employ is the trace norm regularization, which is a popular approach for the estimation of low-rank matrices. In addition, we propose a simple heuristic to improve the interpretability of the obtained factorization. The advantages and disadvantages of three proposed approaches are demonstrated through numerical experiments on both synthetic and real world datasets. We show that the proposed convex optimization based approaches are more accurate in predictive performance, faster, and more reliable in recovering a known multilinear structure than conventional approaches.

Motivation & Objective

• To address the limitations of non-convex optimization in Tucker tensor decomposition, which may converge to poor local minima or stationary points.
• To extend trace norm regularization—successful for low-rank matrices—to multi-way tensors for robust and globally optimal estimation.
• To enable automatic rank estimation without prior specification, improving reliability and reducing user dependency.
• To improve interpretability of the core tensor through a proposed heuristic, especially in real-world applications.
• To demonstrate superior performance in tensor reconstruction from partial observations compared to conventional EM-based approaches.

Proposed method

• The first method, 'as a matrix', treats the tensor as a low-rank matrix after unfolding along a single mode, using trace norm regularization on the matricized tensor.
• The second method, 'constraint', applies trace norm regularization simultaneously across all tensor modes, enforcing low-rank structure in every mode.
• The third method, 'mixture', models the tensor as a sum of K low-rank components, each regularized to be low-rank in its respective mode.
• All three formulations are convex minimization problems, ensuring a unique global minimum and enabling efficient solution via the alternating direction method of multipliers (ADMM).
• A dual problem is derived for each approach, and a duality gap computation is used to monitor convergence during ADMM optimization.
• A heuristic is introduced to improve interpretability by reweighting the core tensor based on singular value decomposition of factor matrices.

Experimental results

Research questions

• RQ1 Can convex optimization with trace norm regularization be effectively extended to multi-way tensors to ensure global convergence and unique solutions?
• RQ2 Can the proposed methods automatically estimate the tensor rank without requiring prior specification?
• RQ3 How do the three proposed approaches—'as a matrix', 'constraint', and 'mixture'—compare in terms of predictive performance and robustness on partially observed tensors?
• RQ4 Does the proposed method achieve a sharp phase transition in reconstruction accuracy at low sampling rates, as observed in matrix completion?
• RQ5 Can the interpretability of the core tensor be meaningfully improved through a simple post-processing heuristic?

Key findings

• The proposed convex optimization-based approaches achieve significantly higher predictive accuracy than conventional EM-based Tucker decomposition on both synthetic and real-world datasets.
• The methods are faster and more reliable in recovering the underlying multilinear structure compared to non-convex alternatives.
• One of the proposed approaches exhibits a sharp threshold behavior: reconstruction transitions from poor to nearly perfect fit at a sampling rate roughly proportional to the sum of the k-ranks of the underlying tensor when dimensions are fixed.
• The 'mixture' approach shows particular promise in balancing flexibility and performance, especially when the tensor is not low-rank in all modes simultaneously.
• The proposed heuristic improves the interpretability of the core tensor, as demonstrated on the amino acid fluorescence dataset.
• The duality gap computation enables reliable convergence monitoring, and the ADMM-based solvers efficiently solve the large-scale optimization problems.