[Paper Review] MDP environments for the OpenAI Gym

Andreas Kirsch|arXiv (Cornell University)|Sep 26, 2017

Reinforcement Learning in Robotics3 references3 citations

TL;DR

This paper presents a Python framework for programmatically specifying simple Markov Decision Processes (MDPs) using a domain-specific language (DSL), enabling researchers to create deterministic and non-deterministic MDP environments with ease. The framework supports visualization, optimal value computation via linear programming, and seamless integration with the OpenAI Gym for testing reinforcement learning agents.

ABSTRACT

The OpenAI Gym provides researchers and enthusiasts with simple to use environments for reinforcement learning. Even the simplest environment have a level of complexity that can obfuscate the inner workings of RL approaches and make debugging difficult. This whitepaper describes a Python framework that makes it very easy to create simple Markov-Decision-Process environments programmatically by specifying state transitions and rewards of deterministic and non-deterministic MDPs in a domain-specific language in Python. It then presents results and visualizations created with this MDP framework.

Motivation & Objective

• To address the difficulty of debugging reinforcement learning agents due to the complexity of existing OpenAI Gym environments.
• To provide a lightweight, extensible way to define simple MDPs programmatically for reproducible testing and validation.
• To enable researchers to verify agent convergence and analyze MDP properties using visualizations and analytical solutions.
• To bridge the gap between abstract MDP theory and practical RL experimentation through a clean, readable DSL in Python.

Proposed method

• Define MDPs using a domain-specific language (DSL) in Python with operator overloading to express state transitions and rewards naturally.
• Support both deterministic and non-deterministic MDPs by allowing multiple outcomes per state-action pair, with optional weights for categorical distributions.
• Implement a conventional Python API as an alternative to the DSL for programmatic MDP specification using methods like `spec.transition()` and `spec.state()`.
• Convert MDPs into networkx graphs for visualization using pydotplus and GraphViz, enabling rendering in Jupyter notebooks.
• Use linear programming to compute optimal value functions (Q-tables and V-vectors) via a dedicated module based on the MDP specification.
• Expose MDPs as OpenAI Gym-compatible environments using the `to_env()` method, supporting rendering as RGB arrays, PNGs, and Jupyter displays.

Experimental results

Research questions

• RQ1 How can MDPs be specified in a human-readable, extensible, and composable way within Python for reinforcement learning research?
• RQ2 Can a DSL-based approach simplify the creation of minimal, reproducible MDP environments for debugging RL algorithms?
• RQ3 To what extent can visualizations and analytical solutions (e.g., optimal value functions) improve the interpretability and testing of RL agents?
• RQ4 How effectively can this framework integrate with the OpenAI Gym ecosystem for end-to-end agent evaluation?
• RQ5 Can this framework support both deterministic and non-deterministic MDPs with uniform and weighted outcome distributions?

Key findings

• The framework successfully enables the specification of MDPs using a clean, readable DSL that supports both deterministic and non-deterministic transitions.
• The DSL allows for natural expression of complex outcome distributions using operator precedence and distributive semantics across state-action combinations.
• MDPs can be visualized as graphs using networkx and GraphViz, with support for rendering in Jupyter notebooks, aiding in debugging and analysis.
• Optimal value functions (Q-tables and V-vectors) are accurately computed using linear programming, providing analytical benchmarks for agent performance.
• The framework generates fully compatible OpenAI Gym environments, including support for rendering in multiple formats (RGB, PNG, Jupyter display).
• Five example MDPs are provided, including four that match the minimal environments in OpenAI Gym's debugging suite, and one novel multi-round non-deterministic MDP.