[Paper Review] A New Approach to Dimensionality Reduction for Anomaly Detection in Data Traffic

Tingshan Huang, Harish Sethu+1 more|arXiv (Cornell University)|Jun 14, 2016

Network Security and Intrusion Detection23 references3 citations

TL;DR

This paper proposes a novel distance-based subspace method for dimensionality reduction in network anomaly detection, dynamically adjusting the number of dimensions per time window based on structural differences between observed and normal traffic covariance matrices. Unlike variance-based PCA, it uses maximum subspace distance to detect anomalies more responsively and accurately, with centralized and distributed algorithm variants validated on real traffic traces.

ABSTRACT

The monitoring and management of high-volume feature-rich traffic in large networks offers significant challenges in storage, transmission and computational costs. The predominant approach to reducing these costs is based on performing a linear mapping of the data to a low-dimensional subspace such that a certain large percentage of the variance in the data is preserved in the low-dimensional representation. This variance-based subspace approach to dimensionality reduction forces a fixed choice of the number of dimensions, is not responsive to real-time shifts in observed traffic patterns, and is vulnerable to normal traffic spoofing. Based on theoretical insights proved in this paper, we propose a new distance-based approach to dimensionality reduction motivated by the fact that the real-time structural differences between the covariance matrices of the observed and the normal traffic is more relevant to anomaly detection than the structure of the training data alone. Our approach, called the distance-based subspace method, allows a different number of reduced dimensions in different time windows and arrives at only the number of dimensions necessary for effective anomaly detection. We present centralized and distributed versions of our algorithm and, using simulation on real traffic traces, demonstrate the qualitative and quantitative advantages of the distance-based subspace approach.

Motivation & Objective

• Address the limitations of variance-based PCA in anomaly detection, such as fixed dimensionality and poor responsiveness to real-time traffic changes.
• Overcome the vulnerability of traditional methods to normal traffic spoofing by focusing on covariance matrix differences between normal and observed traffic.
• Develop a dynamic, adaptive dimensionality reduction approach that selects only the necessary number of dimensions per time window for effective anomaly detection.
• Design both centralized and distributed versions of the algorithm to support scalable deployment in large-scale network monitoring systems.

Proposed method

• Propose a new metric called maximum subspace distance to quantify structural differences between the covariance matrices of normal and observed traffic.
• Formulate a distance-based subspace method that computes the optimal number of dimensions dynamically per time window, rather than using a fixed k as in PCA.
• Use singular value decomposition (SVD) and power iteration to compute the dominant singular value of a projection matrix derived from covariance matrix differences.
• Implement a centralized algorithm (getESD) that computes the maximum subspace distance using iterative SVD-based power methods.
• Design a distributed variant (getESD-D) that enables scalable computation across networked nodes using average consensus and localized power iteration.
• Leverage the fact that anomalies alter traffic feature correlations, making covariance matrix deviations a reliable signal for anomaly detection.

Experimental results

Research questions

• RQ1 Can a dynamic, distance-based approach to dimensionality reduction outperform fixed-k variance-based PCA in detecting network anomalies?
• RQ2 How can the number of reduced dimensions be adaptively selected per time window based on real-time traffic deviations from normal behavior?
• RQ3 To what extent does the maximum subspace distance metric improve anomaly detection sensitivity and reduce false positives compared to traditional PCA?
• RQ4 What are the computational and communication trade-offs of a distributed implementation of the distance-based method in large-scale network monitoring?

Key findings

• The proposed distance-based subspace method dynamically selects the number of dimensions per time window, eliminating the need for a pre-defined variance threshold.
• The method demonstrates superior anomaly detection performance by focusing on changes in covariance structure rather than total variance, improving sensitivity to subtle or spoofed anomalies.
• Simulation results on real traffic traces show that the method achieves higher detection accuracy and lower false positive rates than variance-based PCA.
• The distributed version (getESD-D) maintains high accuracy while scaling efficiently, with computational complexity of O(kpMΔS) and communication complexity of O(kMΔ + kpΔS).
• Theoretical analysis proves that the dominant singular value of the projection matrix increases monotonically with dimension, ensuring convergence and stability of the power iteration method.
• The method is robust to normal traffic spoofing because it detects structural shifts in covariance matrices, not just variance deviations.