[Paper Review] Are Tensor Decomposition Solutions Unique? On the global convergence of HOSVD and ParaFac algorithms

Dijun Luo, Heng Huang+1 more|ArXiv.org|Feb 26, 2009

Tensor decomposition and applications2 references4 citations

TL;DR

This paper investigates the uniqueness and global convergence of tensor decomposition solutions using HOSVD and ParaFac algorithms. Despite the non-convex nature of their objective functions, HOSVD consistently converges to a unique global solution on real-world datasets—even under significant scrambling—while ParaFac does not. The study identifies eigenvalue distribution in correlation matrices as a reliable predictor of HOSVD solution uniqueness.

ABSTRACT

For tensor decompositions such as HOSVD and ParaFac, the objective functions are nonconvex. This implies, theoretically, there exists a large number of local optimas: starting from different starting point, the iteratively improved solution will converge to different local solutions. This non-uniqueness present a stability and reliability problem for image compression and retrieval. In this paper, we present the results of a comprehensive investigation of this problem. We found that although all tensor decomposition algorithms fail to reach a unique global solution on random data and severely scrambled data; surprisingly however, on all real life several data sets (even with substantial scramble and occlusions), HOSVD always produce the unique global solution in the parameter region suitable to practical applications, while ParaFac produce non-unique solutions. We provide an eigenvalue based rule for the assessing the solution uniqueness.

Motivation & Objective

• To investigate the solution uniqueness of tensor decomposition algorithms, particularly HOSVD and ParaFac, in the context of non-convex optimization.
• To determine whether HOSVD and ParaFac produce globally unique solutions across different initializations on real-world and randomized datasets.
• To develop a practical criterion for predicting when HOSVD solutions will be unique, based on eigenvalue distributions of correlation matrices.
• To assess the reliability and repeatability of HOSVD and ParaFac in applications like image compression and retrieval, where solution stability is critical.

Proposed method

• Empirically evaluate HOSVD and ParaFac on six real-world datasets (e.g., AT&T, WANG, Caltech 101) with varying tensor dimensions and ranks.
• Apply three types of data randomization: block scrambling, pixel scrambling (40%, 60%, 80%), and occlusion to test robustness of solutions.
• Compute correlation matrices F, G, H from the tensor data to derive eigenvalue distributions used for uniqueness prediction.
• Use iterative alternating least squares to solve HOSVD and ParaFac, tracking convergence behavior across multiple random initializations.
• Analyze the relationship between eigenvalue clustering (near-equal eigenvalues) and non-uniqueness in HOSVD solutions.
• Compare HOSVD results with GLRAM, observing similar uniqueness patterns, though GLRAM is not detailed due to space constraints.

Experimental results

Research questions

• RQ1 Does HOSVD produce a unique global solution across different initializations on real-world tensor data?
• RQ2 How does data scrambling (block, pixel, occlusion) affect the uniqueness of HOSVD and ParaFac solutions?
• RQ3 Can the eigenvalue distribution of correlation matrices F, G, H predict whether HOSVD will converge to a unique solution?
• RQ4 Why does HOSVD exhibit unique convergence on real data despite non-convex optimization, while ParaFac does not?
• RQ5 Under what conditions does HOSVD fail to converge to a unique solution, and can these be predicted in advance?

Key findings

• On all tested real-world datasets (AT&T, WANG, Caltech 101), HOSVD consistently converged to a unique solution regardless of initialization, indicating high stability and repeatability.
• Even under severe data scrambling (e.g., 80% pixel scramble, block scrambling), HOSVD maintained unique convergence on real data, though not on purely random tensors.
• ParaFac solutions were consistently non-unique across all datasets and initializations, confirming the expected behavior of non-convex optimization.
• HOSVD failed to converge to a unique solution only on highly randomized or purely random tensors, where eigenvalues of F, G, H were nearly identical.
• When multiple eigenmodes had similar eigenvalues (e.g., 8 significant modes with close values), HOSVD became ambiguous in selection, leading to non-unique solutions.
• The eigenvalue distribution of F, G, H matrices reliably predicted HOSVD solution uniqueness: clear separation of top eigenmodes indicated unique convergence, while clustering indicated ambiguity.