[Paper Review] A Dynamic Watermarking Algorithm for Finite Markov Decision Problems

Jiacheng Tang, Jiguo Song+1 more|ArXiv.org|Nov 9, 2021

Smart Grid Security and Resilience29 references4 citations

TL;DR

This paper proposes a dynamic watermarking algorithm for finite Markov decision processes (MDPs) that enhances security against spoofing and deception attacks in control systems. By injecting randomized control policy changes, the method enables CUSUM-based detection of compromised feedback channels, with theoretical bounds on detection delay and false alarm rate, validated through a sensor network power management case study showing trade-offs between security and control performance.

ABSTRACT

Dynamic watermarking, as an active intrusion detection technique, can potentially detect replay attacks, spoofing attacks, and deception attacks in the feedback channel for control systems. In this paper, we develop a novel dynamic watermarking algorithm for finite-state finite-action Markov decision processes and present upper bounds on the mean time between false alarms, and the mean delay between the time an attack occurs and when it is detected. We further compute the sensitivity of the performance of the control system as a function of the watermark. We demonstrate the effectiveness of the proposed dynamic watermarking algorithm by detecting a spoofing attack in a sensor network system.

Motivation & Objective

• To address the growing threat of sensor-based cyberattacks in autonomous vehicles and cyber-physical systems (CPS), particularly spoofing and deception attacks that compromise system safety and reliability.
• To extend dynamic watermarking—previously studied in linear systems—into finite-state, finite-action Markov decision processes (MDPs) to enable detection in more abstract, non-linear control environments.
• To establish theoretical performance bounds on detection delay and false alarm rate for the proposed watermarking scheme in MDPs.
• To quantify the trade-off between system security (attack detection) and control performance (control loss) due to watermark injection.
• To validate the approach through a real-world application in a sensor network power management system under spoofing attack conditions.

Proposed method

• Introduces a dynamic watermarking mechanism that modifies the control policy in an MDP by injecting random perturbations to the action selection process.
• Employs a CUSUM-type detection scheme to monitor the residual signal between expected and observed sensor measurements, leveraging the known statistical properties of the watermark.
• Derives theoretical upper bounds on the mean time between false alarms and mean detection delay using stochastic stability and mixing properties of the MDP process.
• Models the system as a partially observed MDP with latent state, observation, and action processes, and uses the joint state-action-observation process to analyze detection performance.
• Applies the watermarking to a sensor network power management system, simulating spoofing attacks to evaluate detection performance and control loss.
• Uses the stationary distribution of the observation process and empirical frequency estimators to ensure consistent detection under asymptotic conditions.

Experimental results

Research questions

• RQ1 How can dynamic watermarking be adapted to finite Markov decision processes (MDPs) with discrete states and actions?
• RQ2 What are the theoretical upper bounds on the mean time between false alarms and mean detection delay for the proposed watermarking scheme in MDPs?
• RQ3 How does the magnitude of the watermark affect the control performance and system stability in the presence of attacks?
• RQ4 Can the proposed method detect spoofing attacks in real-world sensor network systems with provable performance guarantees?
• RQ5 What is the trade-off between detection sensitivity and control loss when introducing dynamic watermarking into an MDP-based control system?

Key findings

• The proposed dynamic watermarking algorithm achieves provable upper bounds on the mean time between false alarms and mean detection delay, ensuring reliable attack detection in MDPs.
• The method demonstrates effective detection of spoofing attacks in a sensor network power management system, with detection occurring within a bounded time window after the attack onset.
• A trade-off is observed between detection performance and control loss: larger watermark magnitudes improve detection speed but increase control deviation from optimal behavior.
• Theoretical analysis confirms that the detection process is stochastically stable, with the CUSUM detector converging to the correct decision under both normal and attack conditions.
• The stationary distribution of the observation process is shown to be consistently estimated over time, enabling reliable covariance-based detection of anomalies.
• The performance bounds are derived using mixing properties and ergodicity of the MDP process, ensuring robustness under model uncertainty.