[Paper Review] Von Neumann's inequality for tensors

Stéphane Chrétien, Tianwen Wei|arXiv (Cornell University)|Feb 5, 2015

Tensor decomposition and applications5 references3 citations

TL;DR

This paper extends von Neumann's inequality from matrices to higher-order tensors using the Higher-Order Singular Value Decomposition (HOSVD). It establishes that the inner product of two tensors is bounded by the inner product of their mode-wise singular value vectors, and fully characterizes the equality case: equality holds simultaneously across all modes if and only if the tensors' core tensors are proportionally related via orthogonal transformations and permutation matrices applied to their singular spectra.

ABSTRACT

For two matrices in $\mathbb R^{n_1 imes n_2}$, the von Neumann inequality says that their scalar product is less than or equal to the scalar product of their singular spectrum. In this short note, we extend this result to real tensors and provide a complete study of the equality case.

Motivation & Objective

• To generalize von Neumann's inequality, originally valid for matrices, to higher-order real tensors.
• To define a meaningful extension of singular values and singular subspace structure for tensors using the Higher-Order SVD (HOSVD) framework.
• To provide a complete characterization of the equality case in the tensor von Neumann inequality across all modes.
• To enable applications in tensor optimization, particularly in subdifferential analysis of tensor functions.
• To support the use of sparsity-promoting norms as convex surrogates for tensor rank minimization in machine learning and statistics.

Proposed method

• Uses the Higher-Order Singular Value Decomposition (HOSVD) to decompose tensors into orthogonal transformations and a core tensor with orthonormal fibers.
• Defines the mode-$d$ singular spectrum $\sigma^{(d)}(\mathcal{X})$ as the vector of singular values of the mode-$d$ matricization $\mathcal{X}_{(d)}$.
• Applies the classical von Neumann inequality to each mode-$d$ matricization to derive the bound $\langle \mathcal{X}, \mathcal{Y} \rangle \leq \langle \sigma^{(d)}(\mathcal{X}), \sigma^{(d)}(\mathcal{Y}) \rangle$.
• Characterizes equality by analyzing the structure of the core tensors $\mathcal{S}(\mathcal{X})$ and $\mathcal{S}(\mathcal{Y})$ under HOSVD.
• Uses permutation matrices $\Pi_d$ to reorder the singular values so that corresponding blocks in the permuted core tensors are proportional.
• Establishes that equality across all modes holds if and only if there exist orthogonal matrices $W^{(d)}$ such that $\mathcal{D}(\mathcal{X}) = \rho \cdot \mathcal{D}(\mathcal{Y})$ under a common transformation, with $\rho$ constant across blocks.

Experimental results

Research questions

• RQ1 How can von Neumann's inequality for matrices be generalized to higher-order tensors?
• RQ2 What is the appropriate definition of singular values and singular subspaces for tensors to enable such a generalization?
• RQ3 Under what conditions does equality hold in the tensor von Neumann inequality across all modes simultaneously?
• RQ4 How does the structure of the core tensor under HOSVD relate to equality in the inequality?
• RQ5 Can the equality condition be expressed in terms of orthogonal transformations and permutation matrices applied to the singular spectra?

Key findings

• The tensor von Neumann inequality holds: for any two tensors $\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{n_1 \times \cdots \times n_D}$, the inner product satisfies $\langle \mathcal{X}, \mathcal{Y} \rangle \leq \langle \sigma^{(d)}(\mathcal{X}), \sigma^{(d)}(\mathcal{Y}) \rangle$ for each mode $d$.
• Equality holds simultaneously for all modes $d=1,\ldots,D$ if and only if there exist orthogonal matrices $W^{(d)}$ and a tensor $\mathcal{D}(\mathcal{X})$ such that $\mathcal{D}(\mathcal{X}) = \rho \cdot \mathcal{D}(\mathcal{Y})$ under a common transformation involving permutation matrices and orthogonal transformations.
• The equality condition implies that the ranks of the core tensors satisfy $r_x^{(d)} = r_y^{(d)}$ for all $d$, or $r_x^{(d)} > r_y^{(d)}$ for all $d$, or $r_x^{(d)} < r_y^{(d)}$ for all $d$, with the latter two cases requiring proportionality of the core tensor entries.
• When equality holds, the entries of the core tensors $\mathcal{S}(\mathcal{X})$ and $\mathcal{S}(\mathcal{Y})$ are proportional across all indices $i_1,\ldots,i_D$ within the support of the smaller tensor, with the same ratio $\sigma^{(d)}_{i_d}(\mathcal{X}) / \sigma^{(d)}_{i_d}(\mathcal{Y})$ across all modes.
• The equality case is characterized by a block structure in the permuted core tensors, where each block corresponds to a common ratio of singular values, and all entries in a block are proportional between $\mathcal{D}(\mathcal{X})$ and $\mathcal{D}(\mathcal{Y})$.
• The characterization ensures that no entry in $\mathcal{D}(\mathcal{X})$ lies outside the union of these blocks unless it is zero, preserving the proportionality condition.