[Paper Review] Anomaly Detection for Water Treatment System based on Neural Network with Automatic Architecture Optimization

Dmitry Shalyga, Pavel Filonov+1 more|arXiv (Cornell University)|Jul 19, 2018

Anomaly Detection Techniques and Applications31 references62 citations

TL;DR

The paper automatically searches neural network architectures with genetic algorithms to improve anomaly detection on the SWaT water treatment dataset, using NAB as the primary evaluation metric and adding techniques to boost detection quality and interpretability.

ABSTRACT

We continue to develop our neural network (NN) based forecasting approach to anomaly detection (AD) using the Secure Water Treatment (SWaT) industrial control system (ICS) testbed dataset. We propose genetic algorithms (GA) to find the best NN architecture for a given dataset, using the NAB metric to assess the quality of different architectures. The drawbacks of the F1-metric are analyzed. Several techniques are proposed to improve the quality of AD: exponentially weighted smoothing, mean p-powered error measure, individual error weight for each variable, disjoint prediction windows. Based on the techniques used, an approach to anomaly interpretation is introduced.

Motivation & Objective

• Motivate early and accurate anomaly detection in industrial control systems (ICS) data from SWaT.
• Develop an automated method to propose NN architectures tailored to a given time-series dataset.
• Improve detection quality and interpretability through additional techniques and evaluation with NAB.
• Enable anomaly diagnosis by identifying potentially attacked tags corresponding to detected anomalies.

Proposed method

• Represent multivariate time-series data from SWaT with input windows and forecast windows for neural forecasting models.
• Use genetic algorithms to generate and select NN architectures from templates (MLP, CNN, RNN) that minimize MSE on training data.
• Train candidate architectures with backpropagation (Adam optimizer) and evaluate via NAB metric and F1 score.
• Introduce anomaly-enhancing techniques: exponentially weighted smoothing, mean p-powered error, per-tag weights, and disjoint forecasting windows.
• Provide an anomaly interpretation mechanism by locating tags with the greatest prediction errors for detected anomalies.

Experimental results

Research questions

• RQ1 Can automatic neural architecture search improve anomaly detection performance on the SWaT dataset compared to manually chosen models?
• RQ2 What techniques best balance false positives and false negatives in ICS anomaly detection when using NN forecasts?
• RQ3 To what extent can the approach diagnose the specific tags involved in an anomaly and support multi-stage attack interpretation?

Key findings

• The best performing model (MLP template) achieved a NAB score of 69.612 and an F1 score of 0.812.
• The model detected 25 anomalies out of 34 with 7 false positives.
• Anomaly detection had an average delay of 11% of the anomaly length.
• The architecture search found an encoder–analyzer–decoder configuration with specific layer shapes as the top solution.
• GRU-based networks performed better than LSTM-based ones in their comparisons.
• Using a shrinked dataset without nearly periodic tags lowered the F1 score only modestly (6% drop) while NAB reflects early detection emphasis.