[Paper Review] Empirical Evaluation of Four Tensor Decomposition Algorithms

Peter D. Turney|ArXiv.org|Nov 13, 2007

Tensor decomposition and applications24 references31 citations

TL;DR

This paper empirically evaluates four tensor decomposition algorithms—HO-SVD, HOOI, SP, and MP—on time, memory, and reconstruction accuracy across synthetic and real-world tensors. HOOI delivers the highest fit but is limited to small tensors due to high RAM usage, while MP offers the best trade-off for larger tensors, making it preferable when scalability is critical.

ABSTRACT

Higher-order tensor decompositions are analogous to the familiar Singular Value Decomposition (SVD), but they transcend the limitations of matrices (second-order tensors). SVD is a powerful tool that has achieved impressive results in information retrieval, collaborative filtering, computational linguistics, computational vision, and other fields. However, SVD is limited to two-dimensional arrays of data (two modes), and many potential applications have three or more modes, which require higher-order tensor decompositions. This paper evaluates four algorithms for higher-order tensor decomposition: Higher-Order Singular Value Decomposition (HO-SVD), Higher-Order Orthogonal Iteration (HOOI), Slice Projection (SP), and Multislice Projection (MP). We measure the time (elapsed run time), space (RAM and disk space requirements), and fit (tensor reconstruction accuracy) of the four algorithms, under a variety of conditions. We find that standard implementations of HO-SVD and HOOI do not scale up to larger tensors, due to increasing RAM requirements. We recommend HOOI for tensors that are small enough for the available RAM and MP for larger tensors.

Motivation & Objective

• To evaluate the scalability and performance trade-offs of four higher-order tensor decomposition algorithms: HO-SVD, HOOI, SP, and MP.
• To assess how time, memory (RAM and disk), and reconstruction fit vary with tensor size and structure.
• To determine the suitability of each algorithm for large-scale tensor decomposition in real-world applications.
• To validate the use of random tensors in performance evaluation by testing on a real computational linguistics dataset.
• To provide practical recommendations for algorithm selection based on application-specific constraints in time, memory, and accuracy.

Proposed method

• Evaluated four tensor decomposition algorithms: HO-SVD, HOOI, SP, and MP, using both synthetic sparse third- and fourth-order tensors.
• Used standard MATLAB Tensor Toolbox implementations for HO-SVD and HOOI; implemented SP and MP from scratch in MATLAB.
• Measured elapsed run time, RAM and disk space usage, and tensor reconstruction fit (R²-like metric) across varying tensor sizes and core ratios.
• Conducted experiments on random sparse tensors to assess scaling behavior, sensitivity to core size imbalances, and varying input-to-core size ratios.
• Validated findings on a real-world computational linguistics tensor (391×849×1020) derived from word similarity patterns and TOEFL data.
• Used TOEFL word similarity task performance as a real-world metric to correlate fit quality with downstream application success.

Experimental results

Research questions

• RQ1 How do the four tensor decomposition algorithms (HO-SVD, HOOI, SP, MP) scale in terms of time, memory, and reconstruction fit as tensor size increases?
• RQ2 How sensitive is each algorithm’s fit to imbalances in core tensor dimensions, particularly in sparse, higher-order tensors?
• RQ3 Does the performance ranking observed on random tensors hold when evaluated on a real-world computational linguistics dataset?
• RQ4 Can the reconstruction fit of a tensor decomposition predict its effectiveness in a downstream application like word similarity prediction?
• RQ5 What are the practical trade-offs between fit, speed, and memory usage that inform algorithm selection for large-scale tensor decomposition?

Key findings

• HOOI achieved the highest reconstruction fit (22.597) but required 5.77 GiB RAM and was limited to tensors up to 1000³ due to memory constraints.
• MP delivered the second-best fit (22.371) and scaled to 2000³ tensors with only 4.34 GiB RAM, making it suitable for larger datasets.
• SP achieved a fit of 22.321 with low memory usage (4.33 GiB) but was highly sensitive to unbalanced core size ratios, especially in sparse tensors.
• HO-SVD had the lowest fit (21.716) and was outperformed by all other algorithms, with no significant advantage in speed or memory.
• On the real TOEFL dataset, HOOI achieved the highest task accuracy (83.75%), followed by MP and SP (81.25%), while HO-SVD scored only 80.00%.
• The raw tensor (no decomposition) scored 67.50% on the TOEFL task, confirming that tensor decomposition improves performance and validating the use of random tensors in prior experiments.