[Paper Review] Deep Actor-Critic Reinforcement Learning for Anomaly Detection

Chen Zhong, Mustafa Gürsoy+1 more|arXiv (Cornell University)|Aug 28, 2019

Smart Grid Security and Resilience13 references5 citations

TL;DR

This paper proposes a deep actor-critic reinforcement learning framework for active sequential anomaly detection in systems with an unknown number of abnormal processes. By dynamically selecting sensors based on posterior probabilities, the agent minimizes claim delay while maximizing confidence, outperforming the Chernoff test in delay and scalability under high-confidence thresholds.

ABSTRACT

Anomaly detection is widely applied in a variety of domains, involving for instance, smart home systems, network traffic monitoring, IoT applications and sensor networks. In this paper, we study deep reinforcement learning based active sequential testing for anomaly detection. We assume that there is an unknown number of abnormal processes at a time and the agent can only check with one sensor in each sampling step. To maximize the confidence level of the decision and minimize the stopping time concurrently, we propose a deep actor-critic reinforcement learning framework that can dynamically select the sensor based on the posterior probabilities. We provide simulation results for both the training phase and testing phase, and compare the proposed framework with the Chernoff test in terms of claim delay and loss.

Motivation & Objective

• Address the challenge of active sequential anomaly detection when an unknown number of processes can be abnormal simultaneously.
• Design a reinforcement learning agent that dynamically selects sensors to minimize observation delay and maximize decision confidence.
• Overcome limitations of traditional methods like the Chernoff test, which assume distinguishable hypotheses and suffer from high delay in overlapping observation distributions.
• Enable adaptive thresholding in the testing phase to handle varying initial posterior probabilities, especially when starting from a normal state (H₀).

Proposed method

• Formulate the anomaly detection problem as a partially observable Markov decision process (POMDP) with M = 1 + Σₖ₌₁ᴺ (N choose k) hypotheses.
• Use maximum likelihood estimation to approximate unknown observation distributions (Bernoulli with error probability ρ) for normal (f) and abnormal (g) states.
• Implement a deep actor-critic network where the actor selects the next sensor based on posterior probabilities, and the critic evaluates the action-value function.
• Define upper (π_up) and lower (π_low) confidence thresholds to trigger stopping and hypothesis rejection, respectively, with π_up > π_low.
• Train the agent using a reward function that penalizes delay and false decisions, encouraging early stopping under high confidence.
• Refine the training with validation testing to ensure convergence of posterior probabilities to correct hypotheses.

Experimental results

Research questions

• RQ1 How can a reinforcement learning agent dynamically select sensors to minimize claim delay in anomaly detection with an unknown number of abnormal processes?
• RQ2 What is the impact of varying upper and lower confidence thresholds (π_up, π_low) on claim delay and decision loss in the testing phase?
• RQ3 How does the proposed actor-critic framework compare to the Chernoff test in terms of delay and loss when observation distributions are not fully distinguishable?
• RQ4 Can the agent adapt to initial conditions where all processes are normal (H₀), and how does this affect convergence and sampling time?
• RQ5 Under what conditions does the deep RL agent outperform the Chernoff test in terms of both delay and loss?

Key findings

• The proposed actor-critic framework achieves significantly lower claim delay than the Chernoff test, especially when the upper confidence threshold π_up ≥ 0.75.
• As π_up increases, the agent’s decision loss decreases, indicating improved confidence and reduced false positives under high-threshold conditions.
• The Chernoff test exhibits higher delay due to its assumption of fully distinguishable hypotheses, which fails when multiple hypotheses share the same observation distribution (e.g., H₁, H₄, H₅, H₇ all show abnormal behavior in sensor 1).
• When π_low is reduced, the agent’s loss decreases slightly due to stricter rejection criteria, reducing false alarms, but at the cost of increased sampling time.
• The agent demonstrates robustness in the testing phase, requiring more samples when starting from H₀ (high posterior for normal state), but converging reliably under varying initial conditions.
• The framework outperforms the Chernoff test in both delay and loss for high-confidence scenarios (π_up ≥ 0.75), making it more suitable for systems with high sampling costs.