[Paper Review] On Higher-order Singular Value Decomposition from Incomplete Data.

Yangyang Xu|arXiv (Cornell University)|Nov 16, 2014

Tensor decomposition and applications3 citations

TL;DR

This paper proposes a unified optimization framework for higher-order singular value decomposition (HOSVD) on incomplete tensor data, combining tensor imputation and decomposition into a single non-convex problem. By leveraging block coordinate update algorithms with global convergence guarantees, it achieves superior performance over state-of-the-art methods in low-rank tensor completion and applications like face recognition and MRI reconstruction.

ABSTRACT

Abstract. Higher-order singular value decomposition (HOSVD) is an efficient way for data reduction and also eliciting intrinsic structure of multi-dimensional array data. It has been used in many applications, and some of them involve incomplete data. To obtain HOSVD of the data with missing values, one can first impute the missing entries through a certain tensor completion method and then perform HOSVD to the reconstructed data. However, the two-step procedure can be inefficient and does not make reliable decomposition. In this paper, we formulate an incomplete HOSVD problem and combine the two steps into solving a single optimization problem, which simultaneously achieves imputation of missing values and also tensor decomposition. We also present two algorithms for solving the problem based on block coordinate update. Global convergence of both algorithms is shown under mild assumptions. The convergence of the second algorithm implies that of the popular higher-order orthogonality iteration (HOOI) method, and thus we, for the first time, give global convergence of HOOI. In addition, we compare the proposed methods to state-of-the-art ones for solving incomplete HOSVD and also low-rank tensor completion problems and demonstrate the superior performance of our methods over other compared ones. Furthermore, we apply them to face recognition and MRI image reconstruction to show their practical performance. Key words. multilinear data analysis, higher-order singular value decomposition (HOSVD), low-rank tensor completion, non-convex optimization, higher-order orthogonality iteration (HOOI), global convergence.

Motivation & Objective

• Address the inefficiency and unreliability of the traditional two-step approach (imputation followed by HOSVD) for incomplete tensor data.
• Formulate a single optimization problem that jointly performs tensor imputation and higher-order singular value decomposition.
• Develop convergent algorithms for solving the proposed optimization problem under mild assumptions.
• Establish global convergence of the higher-order orthogonality iteration (HOOI) method for the first time.
• Demonstrate improved performance on low-rank tensor completion and real-world applications such as face recognition and MRI reconstruction.

Proposed method

• Formulate an incomplete HOSVD problem as a non-convex optimization problem that jointly minimizes a low-rank approximation error and enforces higher-order orthogonality.
• Introduce two block coordinate update algorithms: one with alternating minimization and one with variable hard thresholding for rank approximation.
• Use tensor tucker decomposition as the underlying structure, with core tensor and factor matrices optimized iteratively.
• Apply alternating least squares updates to factor matrices while fixing the core tensor, and vice versa, ensuring convergence under mild conditions.
• Incorporate a rank-keeping mechanism via hard thresholding to maintain low-rank structure during optimization.
• Prove global convergence of both algorithms under standard assumptions, with the second algorithm's convergence implying that of the HOOI method.

Experimental results

Research questions

• RQ1 Can a unified optimization framework be designed to simultaneously perform tensor imputation and HOSVD on incomplete data, avoiding the inefficiencies of the two-step approach?
• RQ2 Do the proposed block coordinate update algorithms for the unified problem achieve global convergence under mild assumptions?
• RQ3 Does the convergence of the proposed algorithm imply global convergence of the widely used HOOI method, which has previously lacked such a guarantee?
• RQ4 How does the proposed method compare to state-of-the-art methods in low-rank tensor completion tasks in terms of accuracy and efficiency?
• RQ5 Can the proposed method achieve superior performance in real-world applications such as face recognition and MRI image reconstruction?

Key findings

• The proposed unified optimization framework outperforms state-of-the-art methods in low-rank tensor completion tasks, demonstrating higher accuracy and faster convergence.
• The first block coordinate update algorithm converges globally under mild assumptions, providing a theoretical foundation for its use.
• The second algorithm’s convergence implies the global convergence of the HOOI method, resolving a long-standing open question in tensor decomposition.
• The proposed method achieves superior reconstruction accuracy on face recognition and MRI image reconstruction tasks compared to existing approaches.
• Empirical results show that joint imputation and decomposition yield more reliable and robust tensor decompositions than sequential two-step procedures.