[Paper Review] Learning Environment Models with Continuous Stochastic Dynamics

Martin Tappler, Edi Muškardin+2 more|arXiv (Cornell University)|Jun 29, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes CASTLE, a novel automata learning framework that constructs compact, interpretable Markov decision process (MDP) models of continuous stochastic environments by combining dimensionality reduction, clustering, and passive/active learning. The method enables policy synthesis with performance matching or exceeding deep reinforcement learning agents, while providing transparency and analytical usability via formal model checking.

ABSTRACT

Solving control tasks in complex environments automatically through learning offers great potential. While contemporary techniques from deep reinforcement learning (DRL) provide effective solutions, their decision-making is not transparent. We aim to provide insights into the decisions faced by the agent by learning an automaton model of environmental behavior under the control of an agent. However, for most control problems, automata learning is not scalable enough to learn a useful model. In this work, we raise the capabilities of automata learning such that it is possible to learn models for environments that have complex and continuous dynamics. The core of the scalability of our method lies in the computation of an abstract state-space representation, by applying dimensionality reduction and clustering on the observed environmental state space. The stochastic transitions are learned via passive automata learning from observed interactions of the agent and the environment. In an iterative model-based RL process, we sample additional trajectories to learn an accurate environment model in the form of a discrete-state Markov decision process (MDP). We apply our automata learning framework on popular RL benchmarking environments in the OpenAI Gym, including LunarLander, CartPole, Mountain Car, and Acrobot. Our results show that the learned models are so precise that they enable the computation of policies solving the respective control tasks. Yet the models are more concise and more general than neural-network-based policies and by using MDPs we benefit from a wealth of tools available for analyzing them. When solving the task of LunarLander, the learned model even achieved similar or higher rewards than deep RL policies learned with stable-baselines3.

Motivation & Objective

• To address the lack of interpretable, compact, and analyzable models for complex control environments with continuous dynamics.
• To overcome the scalability limitations of traditional automata learning in high-dimensional, continuous state spaces.
• To enable the automatic learning of finite-state MDPs that accurately represent environmental behavior under agent control.
• To support formal verification and analysis of agent decision-making through learned MDPs.
• To provide a scalable, sample-efficient alternative to deep reinforcement learning with transparent, explainable policies.

Proposed method

• Applies dimensionality reduction (e.g., autoencoders) to compress high-dimensional environmental observations into lower-dimensional representations.
• Uses clustering on reduced observations to group similar states and form abstract state equivalence classes.
• Performs passive automata learning on cluster sequences to infer an initial MDP structure with stochastic transitions.
• Employs active sampling via probabilistic model checking to identify and generate trajectories that maximize model refinement.
• Iteratively improves the MDP model by retraining on newly sampled trajectories to increase accuracy and coverage.
• Uses policy synthesis over the learned MDP to guide active sampling, focusing on high-impact state-action sequences.

Experimental results

Research questions

• RQ1 Can automata learning be scaled to environments with continuous, high-dimensional, and stochastic dynamics?
• RQ2 Can a combination of dimensionality reduction and clustering enable effective state abstraction for MDP learning in continuous spaces?
• RQ3 Can active sampling guided by probabilistic model checking significantly improve the accuracy and utility of learned MDPs?
• RQ4 Can policies derived from the learned MDPs solve benchmark control tasks as effectively as deep reinforcement learning agents?
• RQ5 To what extent can the learned MDPs support formal analysis and explanation of agent behavior?

Key findings

• CASTLE successfully learned finite-state MDP models for four OpenAI Gym environments: LunarLander, CartPole, Mountain Car, and Acrobot.
• The learned MDPs were sufficiently accurate to compute policies that solved all control tasks, demonstrating functional fidelity.
• In the LunarLander environment, the computed policy achieved similar or higher cumulative rewards than stable-baselines3 DRL agents, despite not optimizing for rewards.
• The total sample complexity of CASTLE (approx. 490k time steps for Mountain Car) was lower than that of the A2C agent used for comparison (1M time steps).
• The method achieved robust performance across experiments, with minimal sensitivity to hyperparameters like epsilon (0.005) and belief size (4).
• The approach is sample-efficient and scalable, with average experiment runtime between 60–90 minutes, though not yet GPU-accelerated.