[Paper Review] Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections

Csaba Tóth, Patric Bonnier+1 more|arXiv (Cornell University)|Jun 12, 2020

Computational Physics and Python Applications101 references4 citations

TL;DR

This paper introduces Seq2Tens, a method that represents sequences using low-rank tensor projections within a free algebra framework to capture non-commutative order dependencies efficiently. By lifting static feature maps into a structured tensor algebra and applying low-rank approximations, it enables scalable, modular neural network layers that achieve state-of-the-art performance on multivariate time series classification, mortality prediction, and video generation benchmarks.

ABSTRACT

Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies. At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning. We use a classical mathematical object -- the tensor algebra -- to capture such dependencies. To address the innate computational complexity of high degree tensors, we use compositions of low-rank tensor projections. This yields modular and scalable building blocks for neural networks that give state-of-the-art performance on standard benchmarks such as multivariate time series classification and generative models for video.

Motivation & Objective

• To address the challenge of modeling sequential data with complex, non-commutative dependencies due to ordered structure.
• To develop a scalable and modular representation of sequences that preserves structural information without relying on fixed-length vectorization.
• To provide a theoretically grounded, universal feature map for sequences that generalizes classical string and time series methods.
• To enable efficient deep learning architectures for sequence modeling by combining algebraic structure with low-rank tensor approximations.
• To demonstrate the method's effectiveness across discriminative and generative sequence modeling tasks, including time series, video, and text.

Proposed method

• Lifts a static feature map $\phi: \mathcal{X} \to V$ into the free algebra $\mathrm{T}(V)$, which supports a non-commutative product to encode sequence order.
• Constructs a sequence feature map $\Phi(\mathbf{x}) = \prod_{i=1}^L \varphi(\mathbf{x}_i)$ using multiplication in $\mathrm{T}(V)$, where $\varphi$ is a lift of $\phi$.
• Applies low-rank tensor decompositions to the algebraic product to reduce computational complexity while preserving expressive power.
• Introduces modular neural network layers—LS2T and B-LS2T—based on these tensor projections for use in both discriminative and generative models.
• Employs forward imputation instead of zero imputation in preprocessing to stabilize training when using difference operations in models.
• Uses a bidirectional LS2T layer in the encoder of a GP-VAE model to improve global sequence representation, with hyperparameters kept consistent with the original.

Experimental results

Research questions

• RQ1 Can a universal, structured, and efficient representation of sequences be constructed using algebraic tensor products that respect non-commutative order?
• RQ2 How can the computational complexity of free algebra-based sequence representations be reduced while maintaining expressive power?
• RQ3 To what extent can low-rank tensor projections in the free algebra framework improve performance on real-world sequence modeling benchmarks?
• RQ4 Can the proposed method be effectively integrated into both discriminative and generative deep learning models for sequential data?
• RQ5 Does enhancing the encoder's global context modeling via LS2T layers lead to measurable improvements in sequence reconstruction and prediction tasks?

Key findings

• The Seq2Tens method achieves state-of-the-art performance on multivariate time series classification, mortality prediction, and video generation benchmarks.
• In the GP-VAE model, replacing the standard encoder with a B-LS2T layer improved reconstruction performance on HMNIST and Physionet, with a relative improvement in MSE on Sprites when the information bottleneck was alleviated.
• Increasing the width of the first convolutional layer in the encoder from $h=32$ to $h=256$ reduced the information bottleneck and improved MSE to $1.3 \times 10^{-3} \pm 4.9 \times 10^{-5}$ for GP-VAE (B-LS2T), outperforming the baseline.
• The improved performance was attributed to the enhanced encoder capacity allowing the B-LS2T layer’s global modeling benefits to be fully realized.
• Forward imputation was found to be more stable than zero imputation when combined with difference operations in the model, reducing oscillations in missing data regions.
• The method demonstrated broad applicability, serving as a flexible and scalable building block in diverse sequence modeling architectures, including generative models like GP-VAE.