[Paper Review] Anomaly Detection with Tensor Networks

Jinhui Wang, Chase Roberts+2 more|arXiv (Cornell University)|Jun 3, 2020

Anomaly Detection Techniques and ApplicationsComputer Science41 references18 citations

TL;DR

This paper introduces Tensor Network Anomaly Detection (TNAD), a novel linear model that leverages tensor networks to efficiently represent high-dimensional transformations for one-class anomaly detection. By penalizing the Frobenius norm of the transformation matrix, TNAD ensures a tight fit around sparse normal data, outperforming deep and classical methods on tabular datasets and achieving competitive results on images without exploiting spatial locality.

ABSTRACT

Originating from condensed matter physics, tensor networks are compact representations of high-dimensional tensors. In this paper, the prowess of tensor networks is demonstrated on the particular task of one-class anomaly detection. We exploit the memory and computational efficiency of tensor networks to learn a linear transformation over a space with dimension exponential in the number of original features. The linearity of our model enables us to ensure a tight fit around training instances by penalizing the model's global tendency to a predict normality via its Frobenius norm---a task that is infeasible for most deep learning models. Our method outperforms deep and classical algorithms on tabular datasets and produces competitive results on image datasets, despite not exploiting the locality of images.

Motivation & Objective

• To address the challenge of one-class anomaly detection where normal data is abundant but anomalies are rare and span a vast, high-dimensional space.
• To overcome the limitations of deep learning models in controlling behavior across the entire input space due to high model complexity.
• To leverage the memory and computational efficiency of tensor networks for learning compact, expressive representations of high-dimensional feature spaces.
• To ensure a tight fit around normal instances through Frobenius norm regularization of a linear transformation, a strategy infeasible for most deep models.
• To demonstrate that tensor networks can serve as a powerful, scalable alternative to deep learning in anomaly detection without relying on inductive biases like image locality.

Proposed method

• The model uses a linear transformation over a feature space exponentially larger than the input, represented compactly via tensor networks to avoid full-matrix computation.
• A key innovation is the use of Frobenius norm regularization to penalize the model's global tendency toward normality, ensuring a tight decision boundary around training instances.
• The model employs two types of feature embeddings: trigonometric (ϕ_trig) and Fourier-based (ϕ_four), with the latter used for datasets like Glass to better exploit orthogonality.
• Tensor network structure is defined by bond dimension b=5 and a parameter α that controls the embedding dimension relative to the number of training samples.
• A two-phase training scheme is used, with damping and learning rate decay to stabilize optimization, especially for larger models.
• The anomaly score is derived from the model’s decision function, which computes the output of the linear transformation, with higher values indicating greater deviation from normality.

Experimental results

Research questions

• RQ1 Can tensor networks be effectively used to represent high-dimensional linear transformations for anomaly detection in a computationally efficient manner?
• RQ2 Does Frobenius norm regularization of a linear transformation lead to a tighter fit around sparse normal data compared to deep learning models?
• RQ3 Can a linear model with tensor network parameterization outperform both classical and deep learning methods on tabular anomaly detection benchmarks?
• RQ4 How does the performance of a non-locality-aware model like TNAD compare to locality-aware models on image datasets?
• RQ5 To what extent can tensor networks serve as a scalable, wide model architecture for one-class anomaly detection without domain-specific inductive biases?

Key findings

• TNAD achieved the highest AUROC on all five ODDS benchmark datasets, with a 97.3±4.5% score on Wine, significantly outperforming the next best method (OC-SVM at 60.0%).
• On the Glass dataset, TNAD achieved 81.8±7.3% AUROC, surpassing OC-SVM (62.0%) and other baselines, despite the dataset's high class imbalance and small size.
• On the Thyroid dataset, TNAD matched the best baseline (99.0±0.1%) with a score of 99.0±0.1%, demonstrating robustness on moderately sized data.
• On the large Forest Cover dataset, TNAD achieved 98.8±0.6% AUROC, outperforming OC-SVM (97.7%) and all other models, including the self-supervised GOAD (64.6±4.7%).
• GOAD's performance degraded significantly on non-image data like Satellite and Forest Cover, suggesting affine transformations may not generalize well to arbitrary tabular data.
• The use of the Fourier embedding (ϕ_four) on Glass was critical for performance, indicating that embedding design significantly impacts model success on small, high-imbalance datasets.