[Paper Review] Stable Tensor Neural Networks for Rapid Deep Learning

Elizabeth Newman, Lior Horesh+2 more|arXiv (Cornell University)|Nov 15, 2018

Tensor decomposition and applications25 references20 citations

TL;DR

This paper introduces stable tensor neural networks (t-NNs) that leverage the t-product algebra to process multidimensional data more efficiently than traditional matrix-based deep learning. By replacing standard matrix operations with tensor operations, the framework achieves a more compact, powerful parameterization that enables faster learning and improved generalization on MNIST and CIFAR-10 benchmarks.

ABSTRACT

We propose a tensor neural network ($t$-NN) framework that offers an exciting new paradigm for designing neural networks with multidimensional (tensor) data. Our network architecture is based on the $t$-product (Kilmer and Martin, 2011), an algebraic formulation to multiply tensors via circulant convolution. In this $t$-product algebra, we interpret tensors as $t$-linear operators analogous to matrices as linear operators, and hence our framework inherits mimetic matrix properties. To exemplify the elegant, matrix-mimetic algebraic structure of our $t$-NNs, we expand on recent work (Haber and Ruthotto, 2017) which interprets deep neural networks as discretizations of non-linear differential equations and introduces stable neural networks which promote superior generalization. Motivated by this dynamic framework, we introduce a stable $t$-NN which facilitates more rapid learning because of its reduced, more powerful parameterization. Through our high-dimensional design, we create a more compact parameter space and extract multidimensional correlations otherwise latent in traditional algorithms. We further generalize our $t$-NN framework to a family of tensor-tensor products (Kernfeld, Kilmer, and Aeron, 2015) which still induce a matrix-mimetic algebraic structure. Through numerical experiments on the MNIST and CIFAR-10 datasets, we demonstrate the more powerful parameterizations and improved generalizability of stable $t$-NNs.

Motivation & Objective

• Address the high storage and computational costs of deep neural networks with millions of parameters.
• Reduce parameter inefficiency in fully-connected layers by replacing matrices with tensors.
• Develop a tensor-based neural network framework that inherits matrix-like algebraic properties for stable and efficient learning.
• Improve model generalization through a stable forward propagation mechanism inspired by differential equation interpretations.
• Demonstrate the effectiveness of tensor-based parameterization on standard image classification benchmarks.

Proposed method

• Utilize the t-product algebra to define tensor operations that mimic matrix operations, enabling tensor-based linear transformations.
• Replace standard fully-connected layers $ A_{j+1} = \sigma(W_j \cdot A_j + \vec{b}_j) $ with $ \mathcal{A}_{j+1} = \sigma(\mathcal{W}_j * \mathcal{A}_j + \vec{\mathcal{B}}_j) $, where $*$ denotes the t-product.
• Introduce a stable t-NN framework based on the dynamic interpretation of deep networks as discretized non-linear differential equations.
• Generalize the framework to a family of tensor-tensor products that preserve matrix-mimetic algebraic structure.
• Implement back-propagation for a tubal softmax loss function using differentiation in the Fourier domain via the DFT matrix.
• Apply the t-NN architecture to image classification tasks using MNIST and CIFAR-10 datasets to evaluate performance and parameter efficiency.

Experimental results

Research questions

• RQ1 Can tensor-based parameterization reduce the number of parameters in deep neural networks while maintaining or improving performance?
• RQ2 How does the t-product algebra enable matrix-mimetic operations in multidimensional tensor networks?
• RQ3 Does the stable t-NN framework improve generalization and training speed compared to standard matrix-based networks?
• RQ4 To what extent can tensor-tensor products generalize the t-product framework while preserving computational efficiency?
• RQ5 How does the tensor-based approach extract latent multidimensional correlations in image data more effectively than traditional methods?

Key findings

• The stable t-NN framework achieves faster learning due to a more compact and powerful parameterization compared to standard matrix-based networks.
• The t-NN architecture reduces the number of parameters by leveraging multidimensional correlations inherent in tensor data, such as in images and videos.
• Numerical experiments on MNIST and CIFAR-10 show that t-NNs generalize better than comparable matrix-based models, indicating improved robustness.
• The tensor algebra based on the t-product preserves key matrix-like properties, enabling seamless adaptation of matrix-based deep learning techniques to tensor space.
• Back-propagation for the tubal softmax loss is derived via Fourier-domain differentiation, confirming the matrix-mimetic nature of the t-product algebra.
• The framework generalizes to a family of tensor-tensor products that maintain the same algebraic structure and computational advantages.