[Paper Review] Constraints Satisfiability Driven Reinforcement Learning for Autonomous Cyber Defense

Ashutosh Dutta, Ehab Al‐Shaer+1 more|arXiv (Cornell University)|Apr 19, 2021

Smart Grid Security and Resilience9 references4 citations

TL;DR

This paper proposes a hybrid reinforcement learning architecture, Constrained Satisfiability-driven Reinforcement Learning (CSRL), that integrates Satisfiability Modulo Theories (SMT) verification into the decision loop of autonomous cyber defense agents to ensure real-time satisfaction of safety and security constraints. The approach enables fast convergence to optimal defense policies and achieves 99% success in defeating diverse attack strategies in simulated cyber-physical systems.

ABSTRACT

With the increasing system complexity and attack sophistication, the necessity of autonomous cyber defense becomes vivid for cyber and cyber-physical systems (CPSs). Many existing frameworks in the current state-of-the-art either rely on static models with unrealistic assumptions, or fail to satisfy the system safety and security requirements. In this paper, we present a new hybrid autonomous agent architecture that aims to optimize and verify defense policies of reinforcement learning (RL) by incorporating constraints verification (using satisfiability modulo theory (SMT)) into the agent's decision loop. The incorporation of SMT does not only ensure the satisfiability of safety and security requirements, but also provides constant feedback to steer the RL decision-making toward safe and effective actions. This approach is critically needed for CPSs that exhibit high risk due to safety or security violations. Our evaluation of the presented approach in a simulated CPS environment shows that the agent learns the optimal policy fast and defeats diversified attack strategies in 99\% cases.

Motivation & Objective

• To address the lack of real-time, constraint-aware autonomous cyber defense in complex cyber-physical systems (CPS).
• To overcome limitations of static models and unrealistic assumptions in existing reinforcement learning and constraint satisfaction frameworks.
• To develop a hybrid agent architecture that dynamically verifies safety and security constraints during policy learning.
• To enable real-time, adaptive defense planning that remains provably correct under evolving attack strategies and uncertain environments.
• To improve convergence and effectiveness of RL-based defense agents through SMT-driven feedback on constraint satisfiability.

Proposed method

• Formulates the defense problem as a Sequential Decision Process (SDP) with stochastic environment dynamics and mission-oriented constraints.
• Employs model-free reinforcement learning (PPO2 with MlpPolicy) to optimize defense policies through interaction with a simulated environment.
• Introduces a Pre-execution Satisfiability (Pre-sat) module that verifies whether candidate defense actions satisfy all safety and security constraints using SMT solvers.
• Decouples policy optimization from constraint verification, updating the policy based on satisfiability feedback rather than direct constraint enforcement.
• Uses environment feedback to infer implicit or incomplete requirements, improving robustness in dynamic or uncertain domains.
• Implements the framework in Python using OpenAI Gym and Stable Baselines, with two network topologies (100 and 200 devices) for evaluation.

Experimental results

Research questions

• RQ1 How can reinforcement learning be enhanced with formal verification to ensure safety and security constraints in autonomous cyber defense?
• RQ2 To what extent does SMT-based constraint verification improve the convergence and performance of RL-based defense agents in dynamic environments?
• RQ3 Can the agent learn effective defense policies without prior knowledge while dynamically inferring missing or vague requirements from environmental feedback?
• RQ4 How does the integration of Pre-sat verification affect the agent’s ability to defeat diverse attack strategies, including stealthy and aggressive ones?
• RQ5 What is the scalability and real-time feasibility of the proposed hybrid architecture in large-scale CPS environments?

Key findings

• The agent achieved 99% success rate in defeating diverse attack strategies, including stealthy, aggressive, and naive attackers, across both 100- and 200-device topologies.
• The agent converged to optimal defense planning within 50 episodes in the 200-device topology and reached over 87% reward against the stealthy attacker.
• Against aggressive attackers, the agent terminated attack propagation within 25–27 time sequences in 75% of cases, demonstrating fast response times.
• The Pre-sat module improved reward performance by up to 70% against stealthy attackers, showing significant benefit in policy optimization.
• The agent successfully reduced attacker propagation to less than 1% probability of reaching the attack goal state, even in the most challenging scenarios.
• The framework demonstrated scalability and real-time feasibility, with faster convergence in larger topologies due to increased satisfiable planning opportunities.