[Paper Review] Security Analysis of Online Centroid Anomaly Detection

Marius Kloft, Pavel Laskov|arXiv (Cornell University)|Feb 27, 2010

Network Security and Intrusion Detection65 references87 citations

TL;DR

This paper provides a theoretical security analysis of online centroid anomaly detection under adversarial poisoning attacks, deriving bounds on attack effectiveness under various constraints. It shows that while attacks can be highly effective in unconstrained settings, external constraints such as bounded false positive rates or limited data injection can severely limit attacker gains, making attacks arbitrarily difficult to succeed.

ABSTRACT

Security issues are crucial in a number of machine learning applications, especially in scenarios dealing with human activity rather than natural phenomena (e.g., information ranking, spam detection, malware detection, etc.). It is to be expected in such cases that learning algorithms will have to deal with manipulated data aimed at hampering decision making. Although some previous work addressed the handling of malicious data in the context of supervised learning, very little is known about the behavior of anomaly detection methods in such scenarios. In this contribution we analyze the performance of a particular method -- online centroid anomaly detection -- in the presence of adversarial noise. Our analysis addresses the following security-related issues: formalization of learning and attack processes, derivation of an optimal attack, analysis of its efficiency and constraints. We derive bounds on the effectiveness of a poisoning attack against centroid anomaly under different conditions: bounded and unbounded percentage of traffic, and bounded false positive rate. Our bounds show that whereas a poisoning attack can be effectively staged in the unconstrained case, it can be made arbitrarily difficult (a strict upper bound on the attacker's gain) if external constraints are properly used. Our experimental evaluation carried out on real HTTP and exploit traces confirms the tightness of our theoretical bounds and practicality of our protection mechanisms.

Motivation & Objective

• To analyze the vulnerability of online centroid anomaly detection to adversarial poisoning attacks in security-critical applications.
• To formalize the interaction between learning and attack processes in an adversarial setting.
• To derive optimal attack strategies and quantify their effectiveness under bounded and unbounded data injection constraints.
• To investigate how external constraints—such as false positive rate limits—can be leveraged to limit attacker gains.
• To validate theoretical bounds with experiments on real HTTP and exploit traces, confirming tightness and practicality of the protection mechanisms.

Proposed method

• The authors model the learning and attack processes as a stochastic game, where the attacker manipulates data points to maximize deviation from the true centroid.
• They derive bounds on the expected distance between the estimated and true centroid using martingale concentration inequalities and geometric series approximations.
• Key components include the use of Jensen’s inequality, indicator functions for attack success conditions, and recursive expectation bounds on the anomaly score.
• The analysis incorporates constraints such as bounded false positive rates and limited injection percentages, modeling realistic operational limits.
• Theoretical bounds are derived for both first and second moments of the anomaly score, leading to variance bounds that quantify attack resilience.
• The method uses a recursive expectation framework to analyze the evolution of the centroid estimate over time under adversarial influence.

Experimental results

Research questions

• RQ1 How effective can a poisoning attack be against online centroid anomaly detection in the absence of constraints?
• RQ2 What is the optimal attack strategy under bounded data injection and false positive rate constraints?
• RQ3 Can external constraints be used to limit the attacker’s gain to an arbitrarily small value?
• RQ4 How tight are the theoretical bounds in practice, especially under real-world network traffic patterns?
• RQ5 To what extent do the derived protection mechanisms prevent successful adversarial manipulation in real-world traces?

Key findings

• In the unconstrained case, a poisoning attack can be highly effective, with the attacker achieving significant deviation from the true centroid.
• When the false positive rate is bounded, the attacker’s gain is strictly limited by a theoretical upper bound, making successful attacks increasingly difficult.
• With bounded injection percentages, the attacker’s gain is also constrained, and the theoretical bounds closely match empirical results on real HTTP and exploit traces.
• The variance of the anomaly score is bounded by a term that decays with the number of samples, indicating convergence to a stable detection behavior under constraints.
• Experimental evaluation confirms that the theoretical bounds are tight, and protection mechanisms based on constraints are practically effective in real-world scenarios.
• The analysis shows that external constraints such as false positive rate limits can be leveraged to make attacks arbitrarily difficult, even in the presence of adaptive adversaries.