[Paper Review] Formal Language Constraints for Markov Decision Processes

Eleanor Quint, Dong Xu+3 more|arXiv (Cornell University)|Oct 2, 2019

Formal Methods in Verification53 references4 citations

TL;DR

This paper proposes a formal language-based framework for specifying safety constraints in Markov Decision Processes (MDPs) using finite automata to enable efficient runtime verification and safe reinforcement learning. By augmenting MDP states with constraint automaton states, learning dense cost functions, and applying action shaping, the method significantly reduces constraint violations during exploration while improving sample efficiency and final performance across MuJoCo, Safety Gym, and Atari environments.

ABSTRACT

In order to satisfy safety conditions, an agent may be constrained from acting freely. A safe controller can be designed a priori if an environment is well understood, but not when learning is employed. In particular, reinforcement learned (RL) controllers require exploration, which can be hazardous in safety critical situations. We study the benefits of giving structure to the constraints of a constrained Markov decision process by specifying them in formal languages as a step towards using safety methods from software engineering and controller synthesis. We instantiate these constraints as finite automata to efficiently recognise constraint violations. Constraint states are then used to augment the underlying MDP state and to learn a dense cost function, easing the problem of quickly learning joint MDP/constraint dynamics. We empirically evaluate the effect of these methods on training a variety of RL algorithms over several constraints specified in Safety Gym, MuJoCo, and Atari environments.

Motivation & Objective

• To address the challenge of ensuring safety during exploration in reinforcement learning, where uncontrolled exploration can lead to hazardous behaviors.
• To enable formal verification of safety properties by expressing constraints in well-understood formal languages such as regular languages and finite automata.
• To improve sample efficiency and learning stability by structuring constraints to guide exploration and reduce constraint violations.
• To integrate safety constraints into RL training without relying on pre-defined analytical limits or post-hoc shielding.
• To demonstrate that formal language constraints can be efficiently recognized and used to shape rewards and actions during training.

Proposed method

• Formal language constraints are defined using finite automata that recognize valid trajectories, enabling efficient runtime verification of safety conditions.
• The MDP state is augmented with the current state of the constraint automaton to create a joint MDP/constraint state space, improving learning of joint dynamics.
• A dense cost function is learned from a sparse cost signal using a learned reward shaping function, which provides smoother credit assignment during training.
• Action shaping is applied by dynamically restricting the action set to only those actions that do not lead to constraint violations, ensuring safety during exploration.
• The framework integrates with standard RL algorithms (e.g., DQN, SAC) and supports both hard constraints (action filtering) and soft constraints (reward shaping).
• Constraints are specified using regular expressions, such as $(\ell r)^2 \mid (r\ell)^2$, to encode temporal patterns like dithering in Atari and MuJoCo.

Experimental results

Research questions

• RQ1 Can formal language constraints expressed as finite automata improve the safety and sample efficiency of reinforcement learning agents during exploration?
• RQ2 How does augmenting the MDP state with constraint automaton states affect learning convergence and constraint violation rates?
• RQ3 To what extent can learned dense cost functions improve performance compared to sparse cost signals in constrained RL?
• RQ4 Does dynamic action shaping based on constraint structure reduce constraint violations while maintaining high return?
• RQ5 How do formal language constraints compare to traditional shielding or reward shaping in terms of verification support and integration with learning algorithms?

Key findings

• In the Half-cheetah environment, using reward shaping with λ = -1000 reduced violations from 82.84 to 16.73 per 100 steps while increasing average return from 1555.30 to 2524.10.
• For the Reacher environment, the best-performing configuration (λ = -1000) achieved zero violations and a return of -5.28, outperforming the baseline of -6.55.
• In Atari environments, hard action shaping applied during both training and evaluation achieved the highest mean returns (e.g., 302.00 for Breakout, 2284.78 for Seaquest) with zero violations.
• When action shaping was applied only during training, performance improved slightly (e.g., 281.77 for Paddle Ball vs. 229.00 when only in evaluation), suggesting early shaping aids policy learning.
• The method reduced constraint violations by up to 80% in MuJoCo environments while improving final policy returns, demonstrating improved safety and sample efficiency.
• The use of formal language constraints enabled verification of safety properties a priori and supported integration into contract-based software systems.