[Paper Review] Symbolic Generalization for On-line Planning

Zhengzhu Feng, Eric A. Hansen+1 more|arXiv (Cornell University)|Oct 19, 2012

Formal Methods in Verification20 references39 citations

TL;DR

This paper introduces Symbolic Real-Time Dynamic Programming (sRTDP), an on-line planning algorithm that uses symbolic model-checking techniques to generalize experience across groups of states rather than individual states. By dynamically grouping states based on heuristics, sRTDP significantly reduces both computation time and the number of real-world interactions needed for convergence in Markov decision processes.

ABSTRACT

Symbolic representations have been used successfully in off-line planning algorithms for Markov decision processes. We show that they can also improve the performance of on-line planners. In addition to reducing computation time, symbolic generalization can reduce the amount of costly real-world interactions required for convergence. We introduce Symbolic Real-Time Dynamic Programming (or sRTDP), an extension of RTDP. After each step of on-line interaction with an environment, sRTDP uses symbolic model-checking techniques to generalizes its experience by updating a group of states rather than a single state. We examine two heuristic approaches to dynamic grouping of states and show that they accelerate the planning process significantly in terms of both CPU time and the number of steps of interaction with the environment.

Motivation & Objective

• To improve the efficiency of on-line planning in Markov Decision Processes (MDPs) by reducing reliance on individual state updates.
• To decrease the number of real-world interactions required for convergence in practical planning scenarios.
• To extend Real-Time Dynamic Programming (RTDP) with symbolic generalization using model-checking techniques.
• To evaluate heuristic methods for dynamically grouping states to improve planning speed and scalability.

Proposed method

• Extends RTDP by updating groups of states symbolically instead of single states after each environment interaction.
• Employs symbolic model-checking techniques to represent and manipulate sets of states efficiently using binary decision diagrams (BDDs).
• Applies two heuristic approaches to dynamically group states based on similarity in value function or transition structure.
• Uses symbolic generalization to propagate value updates across entire state groups, reducing redundant computations.
• Integrates symbolic abstraction into on-line planning to maintain real-time responsiveness while improving convergence.

Experimental results

Research questions

• RQ1 Can symbolic generalization improve the performance of on-line planning algorithms in MDPs?
• RQ2 How do dynamic state grouping heuristics affect convergence speed and interaction cost in on-line planning?
• RQ3 To what extent can symbolic model-checking reduce computation time and real-world interactions in RTDP?
• RQ4 Does symbolic generalization preserve solution quality while accelerating planning?

Key findings

• sRTDP significantly reduces CPU time compared to standard RTDP by generalizing updates across state groups.
• The number of environment interactions required for convergence is substantially reduced due to symbolic generalization.
• Two heuristic-based dynamic grouping methods accelerate planning, with one showing superior performance in terms of both speed and interaction reduction.
• Symbolic generalization maintains solution quality while enabling scalable on-line planning in complex MDPs.