[Paper Review] On the Nuclear Norm and the Singular Value Decomposition of Tensors

Harm Derksen|arXiv (Cornell University)|Aug 18, 2013

Tensor decomposition and applications11 references4 citations

TL;DR

This paper introduces a novel generalization of the Singular Value Decomposition (SVD) for higher-order tensors, termed the Diagonal Singular Value Decomposition (DSVD), and systematically studies the nuclear norm as a convex relaxation of tensor rank. It establishes that the determinant and permanent tensors do not admit a DSVD for $ n \geq 3 $, and derives exact values for their nuclear norms and spectral norms, revealing fundamental obstructions to low-rank decomposition via this framework.

ABSTRACT

Finding the rank of a tensor is a problem that has many applications. Unfortunately it is often very difficult to determine the rank of a given tensor. Inspired by the heuristics of convex relaxation, we consider the nuclear norm instead of the rank of a tensor. We determine the nuclear norm of various tensors of interest. Along the way, we also do a systematic study various measures of orthogonality in tensor product spaces and we give a new generalization of the Singular Value Decomposition to higher order tensors.

Motivation & Objective

• To develop a new generalization of the SVD for higher-order tensors, termed the Diagonal Singular Value Decomposition (DSVD), that extends the properties of matrix SVD to tensors.
• To study the nuclear norm of tensors as a convex relaxation of tensor rank, motivated by applications in tensor completion and low-rank approximation.
• To determine the nuclear norm and spectral norm for key tensors such as the permanent and determinant, providing exact values and insights into their structural complexity.
• To investigate the existence and limitations of DSVD by analyzing orthogonality conditions and deriving necessary conditions for decomposition.

Proposed method

• Proposes a new tensor decomposition framework, the Diagonal Singular Value Decomposition (DSVD), based on $ 2 $-orthogonal pure tensors with ordered singular values.
• Defines $ t $-orthogonality for $ r $-tuples of pure tensors to generalize orthogonality in tensor product spaces and establish conditions for DSVD existence.
• Uses the nuclear norm as the sum of $ \|v_i\| $ over all decompositions $ T = \sum v_i $, minimizing this sum to define the norm.
• Applies Theorem 1.1 and Corollary 1.2 to derive bounds on spectral norms using duality between nuclear and spectral norms.
• Employs generalized Laplace expansions for the determinant and permanent to derive lower bounds on tensor rank.
• Uses dimension counting and combinatorial arguments to show that $ \|\operatorname{per}_n\|_* = n^{n/2} $, $ \|\det_n\|_* = n! $, and $ [\operatorname{per}_n] = n! / n^{n/2} $.

Experimental results

Research questions

• RQ1 Does the determinant tensor $ \det_n $ admit a Diagonal Singular Value Decomposition (DSVD) for $ n \geq 3 $?
• RQ2 Does the permanent tensor $ \operatorname{per}_n $ admit a DSVD for $ n \geq 3 $?
• RQ3 What are the exact values of the nuclear norm and spectral norm for the permanent and determinant tensors?
• RQ4 What is the relationship between the nuclear norm and spectral norm of a tensor, and how does it constrain the existence of a DSVD?
• RQ5 Can the generalized Laplace expansion be used to derive tight bounds on the tensor rank of $ \det_n $ and $ \operatorname{per}_n $?

Key findings

• The nuclear norm of the permanent tensor $ \operatorname{per}_n $ is exactly $ n^{n/2} $, and its spectral norm is $ n! / n^{n/2} $, so the ratio of nuclear to spectral norm is $ n^n / n! $.
• The determinant tensor $ \det_n $ has nuclear norm $ n! $ and spectral norm $ n! / n^{n/2} $, yielding the same ratio $ n^n / n! $, which is not an integer for $ n \geq 3 $.
• For $ n \geq 3 $, the permanent tensor $ \operatorname{per}_n $ cannot admit a DSVD because the required multiplicity $ r = n^n / n! $ is not an integer.
• For $ n \geq 3 $, the determinant tensor $ \det_n $ cannot admit a DSVD because the required multiplicity $ r = n! $ exceeds the maximum possible number of $ 2 $-orthogonal pure tensors, which is bounded by $ n^{n/2} $.
• The tensor rank of $ \det_3 $ is at most 5, and the rank of $ \det_n $ satisfies $ \operatorname{rank}(\det_n) \leq \left(\frac{5}{6}\right)^{\lfloor n/3 \rfloor} n! $, showing subfactorial growth.
• The generalized Laplace expansion yields a lower bound $ \operatorname{rank}(\det_n) \geq \binom{n}{\lfloor n/2 \rfloor} $, indicating exponential growth in tensor rank.