[Paper Review] Early Anomaly Detection by Learning and Forecasting Behavior.

Tong Zhao, Bo Ni+2 more|arXiv (Cornell University)|Oct 20, 2020

Network Security and Intrusion Detection46 references4 citations

TL;DR

Eland is a novel framework for early graph anomaly detection that enhances performance under data scarcity by jointly learning behavior prediction and anomaly detection through behavior data augmentation. It uses a Seq2Seq-based predictor to forecast user behaviors, which are then used to augment the behavior graph, improving downstream anomaly detection accuracy even with limited early-stage data.

ABSTRACT

Graph anomaly detection systems aim at identifying suspicious accounts or behaviors on social networking sites and e-commercial platforms. Detecting anomalous users at an early stage is crucial to minimize financial loss. When a great amount of observed behavior data are available, existing methods perform effectively though it may have been too late to avoid the loss. However, their performance would become unsatisfactory when the observed data are quite limited at the early stage. In this work, we propose Eland, a novel framework that uses behavior data augmentation for early anomaly detection. It has a Seq2Seq-based behavior predictor that predicts (i) whether a user will adopt a new item or an item that has been historically adopted and (ii) which item will be adopted. Eland exploits the mutual enhancement between behavior prediction and graph anomaly detection. The behavior graph is augmented with the predicted behaviors such that the graph-based anomaly detection methods can achieve better performance, and the detection results can support the behavior predictor in return. Experiments show that Eland improves the performance of a variety of graph-based anomaly detection methods. With the augmented methods in Eland, the performance of anomaly detection at an earlier stage is comparable with or better than non-augmented methods on a greater amount of observation.

Motivation & Objective

• Address the challenge of poor performance in graph anomaly detection when behavior data are limited at early stages.
• Improve early detection by leveraging predictive modeling of user behaviors to augment the behavior graph.
• Enable mutual enhancement between behavior prediction and anomaly detection for improved robustness and accuracy.
• Reduce financial loss by detecting anomalous users earlier, before significant damage occurs.

Proposed method

• Employ a Seq2Seq-based behavior predictor to forecast whether a user will adopt a new or previously adopted item and which specific item will be adopted.
• Augment the original behavior graph with predicted behaviors to enrich the graph structure for better anomaly detection.
• Integrate the behavior predictor and graph-based anomaly detection model in a feedback loop, where detection results refine behavior predictions.
• Use the enhanced graph structure to improve the performance of existing graph-based anomaly detection methods.
• Leverage mutual enhancement: improved behavior prediction strengthens anomaly detection, and anomaly detection results refine behavior predictions.
• Apply the framework to real-world social and e-commercial platforms to evaluate early-stage detection under data scarcity.

Experimental results

Research questions

• RQ1 Can behavior prediction improve early graph anomaly detection when observed data are limited?
• RQ2 How does data augmentation via predicted behaviors enhance the performance of graph-based anomaly detection?
• RQ3 To what extent does the mutual enhancement between behavior prediction and anomaly detection improve detection accuracy?
• RQ4 Can Eland achieve detection performance comparable to or better than non-augmented methods with significantly less observed data?

Key findings

• Eland improves the performance of various graph-based anomaly detection methods through behavior data augmentation.
• With augmented data, Eland achieves detection performance at earlier stages that is comparable to or better than non-augmented methods with more observation data.
• The joint learning of behavior prediction and anomaly detection leads to mutual enhancement, improving both components.
• The framework is effective even when training data are scarce, making it suitable for early anomaly detection in real-world systems.