[Paper Review] SPUDD: Stochastic Planning using Decision Diagrams

Jesse Hoey, Robert St‐Aubin+2 more|arXiv (Cornell University)|Jan 23, 2013

Bayesian Modeling and Causal InferenceComputer Science18 references387 citations

TL;DR

SPUDD proposes a value iteration algorithm for Markov decision processes (MDPs) that uses algebraic decision diagrams (ADDs) to compactly represent value functions and policies, enabling efficient planning in large MDPs. By directly applying dynamic programming to ADDs and Bayesian network representations, the method achieves up to a thirty-fold reduction in nodes needed to represent optimal value functions compared to tree-structured methods, demonstrating significant scalability on problems with up to 63 million states.

ABSTRACT

Markov decisions processes (MDPs) are becoming increasing popular as models of decision theoretic planning. While traditional dynamic programming methods perform well for problems with small state spaces, structured methods are needed for large problems. We propose and examine a value iteration algorithm for MDPs that uses algebraic decision diagrams(ADDs) to represent value functions and policies. An MDP is represented using Bayesian networks and ADDs and dynamic programming is applied directly to these ADDs. We demonstrate our method on large MDPs (up to 63 million states) and show that significant gains can be had when compared to tree-structured representations (with up to a thirty-fold reduction in the number of nodes required to represent optimal value functions).

Motivation & Objective

• To address the scalability limitations of traditional dynamic programming in large-state-space MDPs.
• To develop a structured representation method that efficiently encodes value functions and policies in high-dimensional decision problems.
• To enable efficient value iteration in MDPs using algebraic decision diagrams (ADDs) and Bayesian networks.
• To demonstrate significant memory and computational advantages over tree-structured representations in large-scale planning problems.

Proposed method

• The method represents MDPs using Bayesian networks to model state transitions and rewards.
• Value functions and policies are encoded using algebraic decision diagrams (ADDs), enabling compact, symbolic manipulation.
• Dynamic programming operations—such as value iteration and Bellman backups—are applied directly to the ADD structures.
• The algorithm performs operations like minimization and maximization on ADDs to compute optimal policies without explicit state enumeration.
• ADDs are updated iteratively during value iteration, maintaining a compact representation of the value function across all states.
• The approach leverages the sparsity and structure of MDPs by exploiting shared substructures in the ADDs to reduce computational cost.

Experimental results

Research questions

• RQ1 Can algebraic decision diagrams (ADDs) effectively represent value functions and policies in large MDPs with minimal memory overhead?
• RQ2 How does ADD-based dynamic programming compare to traditional tree-structured representations in terms of memory efficiency and scalability?
• RQ3 To what extent can ADDs preserve the structure of MDPs to enable efficient value iteration without explicit state enumeration?
• RQ4 Can the method scale to MDPs with tens of millions of states while maintaining computational tractability?
• RQ5 What is the performance gain in terms of node count reduction when using ADDs versus tree-structured representations?

Key findings

• The SPUDD method achieved up to a thirty-fold reduction in the number of nodes required to represent optimal value functions compared to tree-structured representations.
• The approach successfully scaled to MDPs with up to 63 million states, demonstrating feasibility for large-scale planning problems.
• ADD-based representation enabled efficient dynamic programming by avoiding explicit enumeration of all states.
• The method maintained high accuracy in policy computation, even in complex, high-dimensional domains.
• Significant memory savings were observed due to the compact encoding of shared substructures in the ADDs.
• The results show that structured representation using ADDs is a viable and efficient alternative to traditional value iteration in large MDPs.