[Paper Review] On the Complexity of Solving Markov Decision Problems

Michael L. Littman, Thomas Dean+1 more|arXiv (Cornell University)|Feb 20, 2013

Reinforcement Learning in Robotics34 references389 citations

TL;DR

This paper analyzes the computational complexity of solving Markov Decision Problems (MDPs), establishing that while MDPs are solvable in polynomial time in theory, practical algorithms for large-scale problems remain limited. It advocates for new analytical approaches based on MDP structure to improve efficiency and scalability in reinforcement learning and automated planning applications.

ABSTRACT

Markov decision problems (MDPs) provide the foundations for a number of problems of interest to AI researchers studying automated planning and reinforcement learning. In this paper, we summarize results regarding the complexity of solving MDPs and the running time of MDP solution algorithms. We argue that, although MDPs can be solved efficiently in theory, more study is needed to reveal practical algorithms for solving large problems quickly. To encourage future research, we sketch some alternative methods of analysis that rely on the structure of MDPs.

Motivation & Objective

• To clarify the theoretical and practical complexity of solving Markov Decision Problems (MDPs) in AI and reinforcement learning.
• To identify gaps between polynomial-time theoretical solvability and the inefficiency of existing algorithms on large-scale problems.
• To motivate the development of more efficient algorithms by analyzing structural properties of MDPs.
• To encourage research into alternative algorithmic analysis methods that leverage MDP structure for improved performance.
• To provide a foundation for future work on scalable and efficient MDP solution techniques in automated planning and reinforcement learning.

Proposed method

• Analyzes the computational complexity of standard MDP solution algorithms, such as value iteration and policy iteration.
• Demonstrates that MDPs are solvable in polynomial time under standard assumptions, establishing theoretical tractability.
• Highlights the disconnect between theoretical efficiency and practical performance on large or complex MDPs.
• Proposes that algorithmic analysis should shift toward exploiting structural features of MDPs, such as state space sparsity or transition patterns.
• Encourages the use of problem-specific structural insights to design faster, more scalable solvers.
• Reviews existing algorithms and their running times, emphasizing the need for new complexity analysis frameworks beyond worst-case bounds.

Experimental results

Research questions

• RQ1 Why do theoretically efficient MDP algorithms often perform poorly on large-scale problems in practice?
• RQ2 What structural properties of MDPs can be exploited to design faster and more scalable solution algorithms?
• RQ3 How can complexity analysis of MDPs be extended beyond worst-case polynomial bounds to reflect real-world performance?
• RQ4 What alternative algorithmic frameworks can improve the efficiency of MDP solving while maintaining optimality guarantees?
• RQ5 In what ways can the inherent structure of MDPs inform the design of practical reinforcement learning and planning systems?

Key findings

• MDPs are solvable in polynomial time, confirming their theoretical tractability under standard assumptions.
• Despite polynomial-time solvability, existing algorithms often exhibit poor performance on large or complex MDPs in practice.
• The gap between theoretical efficiency and practical scalability indicates a need for new algorithmic approaches.
• Structural properties of MDPs—such as sparsity in transitions or state space—can be leveraged to design faster solvers.
• Current complexity analysis frameworks are insufficient for predicting real-world performance, necessitating new analytical methods.
• The paper calls for research into structure-aware algorithms to close the gap between theory and practice in MDP solving.