[Paper Review] Tensor Regression Networks

Jean Kossaifi, Zachary C. Lipton+4 more|arXiv (Cornell University)|Jul 26, 2017

Tensor decomposition and applications49 references78 citations

TL;DR

The paper introduces Tensor Contraction Layers (TCLs) and Tensor Regression Layers (TRLs) to preserve multilinear structure in neural networks, enabling large parameter reductions with competitive accuracy on ImageNet and improved MRI-based trait predictions.

ABSTRACT

Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.

Motivation & Objective

• Preserve multilinear structure in activation tensors throughout CNNs instead of flattening before fully connected layers.
• Introduce TCLs to compress activations with tensor contractions.
• Introduce TRLs to model outputs via low-rank multilinear mappings without flattening.
• Show parameter efficiency and accuracy trade-offs on large-scale and medical imaging datasets.

Proposed method

• Define and integrate Tensor Contraction Layers (TCLs) that map activation tensors X to a core G via X' = X ×1 V(0) ×2 V(1) ... ×N+1 V(N).
• Define Tensor Regression Layers (TRLs) that learn a low-rank Tucker-structured weight tensor W = ⟪G; U(0),...,U(N),U(N+1)⟫ and compute Y = ⟨X, W⟩N + b.
• Derive gradient expressions for TCLs and TRLs to enable end-to-end backpropagation.
• Show equivalence of TCLs to fully connected layers via tensor product view and emphasize parameter count reduction (sum vs product of dimensions).
• Provide efficient implementation by rewriting Y in terms of低-rank subspaces to minimize high-dimensional computation.
• Discuss regularization via low-rank constraints and normalization on factor matrices.

Experimental results

Research questions

• RQ1 Can preserving multimodal tensor structure through TCLs and TRLs match or exceed fully connected layers on large-scale vision tasks?
• RQ2 To what extent do TCLs and TRLs reduce parameter counts while maintaining accuracy on ImageNet?
• RQ3 Do TRLs offer advantages for structure-rich medical imaging data (e.g., MRI) over traditional flattening approaches?
• RQ4 How does end-to-end training of tensorized architectures compare to traditional CNNs in terms of performance and regularization?
• RQ5 What is the practical efficiency gain when implementing tensor contractions in modern hardware?

Key findings

• On ImageNet with ResNet-101, replacing the FC layer with a TRL yields similar or better Top-1/Top-5 accuracy with substantial space savings (e.g., 25% up to 92.4% savings) compared to the baseline.
• Smaller TRL configurations maintain or improve accuracy while achieving large parameter reductions (e.g., up to ~65% space savings with minimal accuracy loss).
• TRLs combined with TCLs preserve multilinear structure and reduce parameters by replacing flatten+FC with tensor-based mappings.
• In MRI-based UK Biobank tasks (Age, Gender, BMI), TRLs significantly outperform a baseline 3D-ResNet FC setup, achieving MAE reductions (Age: 2.96→2.70 years), Gender error reduction (0.79%→0.53%), and BMI MAE reduction (2.37→2.26).
• The results demonstrate that tensor-structured networks can leverage topological data properties, particularly in medical imaging, to improve predictive performance.