[Paper Review] A Notation for Markov Decision Processes

Philip S. Thomas, Okal, Billy|arXiv (Cornell University)|Dec 30, 2015

Reinforcement Learning in Robotics1 references12 citations

TL;DR

This paper introduces MDPNv1, a standardized notation for Markov Decision Processes (MDPs) that enables researchers to replace lengthy notational definitions with a single sentence: 'We use the notational standard MDPNv1.' The notation unifies core MDP components—states, actions, rewards, transition and reward functions—using consistent symbols and provides a LaTeX style file for easy integration. It streamlines academic writing by reducing redundancy and enhancing clarity across reinforcement learning papers.

ABSTRACT

This paper specifies a notation for Markov decision processes.

Motivation & Objective

• To reduce redundancy in reinforcement learning papers by replacing multiple paragraphs of notational definitions with a single standardized notation.
• To provide a common, consistent foundation for MDP notation across RL research, minimizing notational inconsistency and improving readability.
• To enable authors to easily adopt and customize the notation using a LaTeX style file, supporting multiple notational variants without manual equation editing.
• To allow authors to override or extend MDPNv1 definitions when needed, ensuring flexibility without sacrificing standardization.
• To support both discrete and continuous random variables through a controlled abuse of notation, preserving mathematical rigor while maintaining readability.

Proposed method

• Proposes MDPNv1 as a tuple-based MDP definition: $(\mathcal{S},\mathcal{A},\mathcal{R},P,R,d_{0},\gamma)$, with standardized symbols for states, actions, rewards, and functions.
• Defines state, action, and reward as random variables $S_t$, $A_t$, $R_t$, with $s$, $a$, $r$ as their realizations.
• Introduces $P(s,a,s') = \Pr(S_{t+1}=s' \mid S_t=s, A_t=a)$ as the transition function, with three alternative notations: $P(s'\mid s,a)$, $P_s^a(s')$, and $P_{s,s'}^a$.
• Defines the reward function $R(s,a,s',r) = \Pr(R_t = r \mid S_t=s, A_t=a, S_{t+1}=s')$ to model reward distributions conditioned on state transitions.
• Provides a LaTeX style file with commands like \sset, \aset, \rset, \T, \R, \pp, and \mu that generate standardized notation and support subscripts/superscripts.
• Allows authors to override MDPNv1 definitions via later declarations (e.g., redefining $\mathcal{A}$ as advantage functions) while preserving backward compatibility.

Experimental results

Research questions

• RQ1 How can the redundancy of defining MDP components in every reinforcement learning paper be reduced?
• RQ2 What standardized notation can unify MDP representation across RL research while remaining flexible for customization?
• RQ3 How can a consistent, reusable notation be implemented in LaTeX to minimize manual equation editing?
• RQ4 What trade-offs arise when treating continuous or mixed random variables as if they were discrete in notation?
• RQ5 How can authors safely override or extend a standardized notation without breaking consistency?

Key findings

• The adoption of MDPNv1 reduces the need to redefine core MDP components in each paper, saving space and improving clarity.
• The notation supports both discrete and continuous random variables through a controlled abuse of notation, treating all distributions as probability mass functions despite technical inaccuracy in continuous cases.
• Authors can override MDPNv1 definitions (e.g., redefining $\mathcal{A}$) by declaring new meanings later in the paper, preserving flexibility.
• The LaTeX style file enables dynamic switching between notational variants (alpha, beta, kappa) and supports standard LaTeX formatting like subscripts and superscripts.
• Including $\mathcal{R}$ explicitly avoids ambiguity in summations over rewards, resolving issues with implicit definitions like $\sum_{r \in \mathbb{R}}$ or $\sum_{r \in \mathbb{Z}}$.
• The notation is designed to be minimal and extensible, focusing only on essential MDP components to avoid controversy and unnecessary complexity.