[Paper Review] Kernelized Support Tensor Train Machines

Cong Chen, Kim Batselier+2 more|arXiv (Cornell University)|Jan 2, 2020

Tensor decomposition and applications30 references4 citations

TL;DR

This paper introduces the first kernelized support tensor train machine (K-STTM), which leverages tensor train (TT) decomposition to enable kernel methods on multi-dimensional data. By preserving the TT structure in feature space and allowing different kernel functions per tensor mode, K-STTM achieves superior performance in few-sample, high-dimensional image classification tasks, outperforming linear tensor methods and standard SVMs on MNIST, CIFAR-10, and fMRI datasets.

ABSTRACT

Tensor, a multi-dimensional data structure, has been exploited recently in the machine learning community. Traditional machine learning approaches are vector- or matrix-based, and cannot handle tensorial data directly. In this paper, we propose a tensor train (TT)-based kernel technique for the first time, and apply it to the conventional support vector machine (SVM) for image classification. Specifically, we propose a kernelized support tensor train machine that accepts tensorial input and preserves the intrinsic kernel property. The main contributions are threefold. First, we propose a TT-based feature mapping procedure that maintains the TT structure in the feature space. Second, we demonstrate two ways to construct the TT-based kernel function while considering consistency with the TT inner product and preservation of information. Third, we show that it is possible to apply different kernel functions on different data modes. In principle, our method tensorizes the standard SVM on its input structure and kernel mapping scheme. Extensive experiments are performed on real-world tensor data, which demonstrates the superiority of the proposed scheme under few-sample high-dimensional inputs.

Motivation & Objective

• Address the limitation of standard SVMs in handling tensorial data directly, especially in few-sample, high-dimensional settings.
• Overcome the overfitting and computational inefficiency of vectorized approaches by exploiting the intrinsic multi-way structure of tensor data.
• Develop a kernelized framework that preserves the tensor train (TT) structure in feature space while maintaining kernel consistency and information preservation.
• Enable flexible, mode-specific kernel functions to capture heterogeneous data characteristics across tensor modes.
• Demonstrate the superiority of kernelized tensor methods over linear tensor learning and vector-based SVMs in low-data regimes.

Proposed method

• Propose a TT-based feature mapping that maintains the tensor train structure in the high-dimensional feature space, enabling nonlinear classification.
• Design two TT-consistent kernel construction methods: one preserving inner product structure and another maximizing information retention.
• Allow different kernel functions (e.g., RBF, polynomial, linear) to be applied independently on each tensor mode, exploiting mode-specific data characteristics.
• Construct the kernel matrix using TT-based operations to ensure compatibility with the TT decomposition and reduce computational cost.
• Integrate the TT-based kernel into a support vector machine framework, forming the kernelized support tensor train machine (K-STTM).
• Use alternating optimization and low-rank TT approximations to scale the method to high-dimensional data while maintaining model efficiency.

Experimental results

Research questions

• RQ1 Can a kernelized learning framework be successfully built on tensor train decompositions to handle multi-dimensional data while preserving structural and information properties?
• RQ2 Does applying different kernel functions to different tensor modes improve classification performance compared to using a single kernel across all modes?
• RQ3 How does the proposed K-STTM method compare to linear tensor learning models and standard SVMs in terms of accuracy and robustness under few-sample, high-dimensional conditions?
• RQ4 Can the TT-based kernel construction maintain kernel consistency and information preservation while enabling nonlinear classification?
• RQ5 What is the impact of mode-specific kernel selection on classification performance in real-world image datasets?

Key findings

• K-STTM-Prod achieved the highest classification accuracy on all subjects in the fMRI dataset, outperforming SVM, 3D CNN, and linear tensor methods.
• On the MNIST dataset, K-STTM-Prod achieved a test accuracy of 98.2% with only 100 training samples per class, significantly outperforming linear models and standard SVM.
• On CIFAR-10, applying different kernels (RBF on spatial modes, linear/polynomial on color mode) improved classification accuracy over the baseline RBF-RBF-RBF configuration on 9 out of 10 class pairs.
• The TT-based kernel construction preserved information and maintained consistency with the TT inner product, enabling stable and efficient kernel computation.
• The method demonstrated robustness in low-data regimes, where conventional methods like DuSK and 3D CNN failed due to high dimensionality and insufficient samples.
• The ability to apply different kernels per mode led to measurable performance gains, validating the importance of mode-specific modeling in tensor-based learning.