[Paper Review] Tensor Valued Common and Individual Feature Extraction: Multi-dimensional Perspective

Ilia Kisil, Giuseppe G. Calvi+1 more|arXiv (Cornell University)|Nov 1, 2017

Tensor decomposition and applications19 references3 citations

TL;DR

This paper proposes a tensor-based framework for extracting common and individual features from multi-block data using CPD and LL1 tensor decompositions with non-negativity constraints. By exploiting the multi-linear structure of data, it reduces dimensionality and enhances classification accuracy—demonstrated by up to 94.7% accuracy on the ORL face dataset using SVM, significantly outperforming original data and matrix-based methods.

ABSTRACT

A novel method for common and individual feature analysis from exceedingly large-scale data is proposed, in order to ensure the tractability of both the computation and storage and thus mitigate the curse of dimensionality, a major bottleneck in modern data science. This is achieved by making use of the inherent redundancy in so-called multi-block data structures, which represent multiple observations of the same phenomenon taken at different times, angles or recording conditions. Upon providing an intrinsic link between the properties of the outer vector product and extracted features in tensor decompositions (TDs), the proposed common and individual information extraction from multi-block data is performed through imposing physical meaning to otherwise unconstrained factorisation approaches. This is shown to dramatically reduce the dimensionality of search spaces for subsequent classification procedures and to yield greatly enhanced accuracy. Simulations on a multi-class classification task of large-scale extraction of individual features from a collection of partially related real-world images demonstrate the advantages of the "blessing of dimensionality" associated with TDs.

Motivation & Objective

• To address the curse of dimensionality in large-scale, multi-block data by leveraging tensor structures.
• To extract physically meaningful common and individual features from multi-block data using tensor decompositions.
• To improve classification performance by reducing data variance and dimensionality through low-rank tensor approximations.
• To ensure mathematical tractability and interpretability by imposing non-negativity constraints on factor matrices.
• To demonstrate the superiority of tensor-based feature extraction over traditional matrix-based approaches in real-world classification tasks.

Proposed method

• The method uses multi-linear rank decomposition (LL1) and CANDECOMP/PARAFAC (CPD) to separate common and individual feature subspaces from tensor-structured data.
• It imposes non-negativity constraints on the mode-3 factor matrix to preserve physical interpretability, especially in image data.
• The framework leverages the outer product structure in tensor decompositions to model shared information across multiple observations.
• An alternating least squares (ALS) algorithm is adapted for CPD and LL1 with non-negative least squares optimization to enforce physical constraints.
• Data is tensorized by stacking images of the same subject along the third mode, forming a 3D tensor for joint analysis.
• Classification is performed on extracted individual features using standard models (SVM, NN, QD, cKNN), with evaluation on original test images for fairness.

Experimental results

Research questions

• RQ1 How can common and individual features be effectively separated from multi-block data using tensor decompositions?
• RQ2 What is the role of the outer product in linking tensor factorization to physically meaningful feature extraction?
• RQ3 How does imposing non-negativity on factor matrices improve interpretability and performance in image feature extraction?
• RQ4 To what extent does tensor-based feature extraction reduce dimensionality and improve classification accuracy compared to matrix-based methods?
• RQ5 Can the proposed method generalize across different data structures and classification models without requiring uniform input dimensions?

Key findings

• The proposed method achieved 94.7% classification accuracy using SVM on individual features extracted via LL1 decomposition, significantly outperforming the original data (83.9%).
• Neural Network (NN) classification accuracy improved from 4.35% on original data to 92.2% using extracted individual features, indicating reduced overfitting.
• The LL1-based method yielded the highest accuracy across all classifiers, demonstrating its effectiveness in extracting discriminative features.
• The non-negativity constraint enhanced physical interpretability and contributed to lower variance in the extracted features, improving model generalization.
• The framework generalizes well to tensors of any order, provided observations are concatenated along the same mode, enabling flexible application to diverse data types.
• The results confirm the 'blessing of dimensionality' in tensor decompositions, where high-dimensional structure enables more efficient and accurate feature extraction.