[Paper Review] Planning by Prioritized Sweeping with Small Backups

Harm van Seijen, Richard S. Sutton|arXiv (Cornell University)|Jan 10, 2013

Reinforcement Learning in Robotics5 references24 citations

TL;DR

This paper introduces small backups—fine-grained, single-successor-state value updates that reduce computation time per backup to O(1), independent of successor count. By enabling more frequent, targeted updates, prioritized sweeping with small backups achieves significantly better sample efficiency than classical methods, outperforming both Moore & Atkeson and Peng & Williams implementations, even with only one update cycle per time step.

ABSTRACT

Efficient planning plays a crucial role in model-based reinforcement learning. Traditionally, the main planning operation is a full backup based on the current estimates of the successor states. Consequently, its computation time is proportional to the number of successor states. In this paper, we introduce a new planning backup that uses only the current value of a single successor state and has a computation time independent of the number of successor states. This new backup, which we call a small backup, opens the door to a new class of model-based reinforcement learning methods that exhibit much finer control over their planning process than traditional methods. We empirically demonstrate that this increased flexibility allows for more efficient planning by showing that an implementation of prioritized sweeping based on small backups achieves a substantial performance improvement over classical implementations.

Motivation & Objective

• To address the high computational cost of full backups in value iteration and prioritized sweeping, which scale with the number of successor states.
• To develop a more efficient planning mechanism that allows finer control over computation time allocation.
• To enable effective planning under tight computational constraints, particularly in real-time or resource-limited environments.
• To demonstrate that small backups can outperform classical full-backup-based prioritized sweeping methods in sample efficiency and convergence speed.

Proposed method

• Introduces the small backup operation: A ← A − x_j + X_j, where only the changed value of a single successor state X_j is updated, rather than recomputing the full sum.
• Applies small backups within a prioritized sweeping framework, where states are prioritized based on expected value change magnitude.
• Uses a priority queue to select the next state to update, ensuring high-impact value changes are propagated first.
• Employs a model-based approach with stored transition probabilities and rewards, allowing backward propagation of value changes without environment interaction.
• Implements a parameter-free method by using small backups instead of sample backups, avoiding the need for step-size hyperparameter tuning.
• Introduces optimism in the face of uncertainty by initializing unvisited state-action pairs with optimistic values (e.g., 0) until visited M times.

Experimental results

Research questions

• RQ1 Can a backup mechanism that updates only a single successor state achieve better sample efficiency than full backups in planning?
• RQ2 Does reducing the per-backup computation cost enable more frequent and targeted value updates, improving convergence speed?
• RQ3 Can small backups support a parameter-free planning method that matches the performance of TD(0) without requiring step-size tuning?
• RQ4 How does the performance of prioritized sweeping with small backups compare to classical implementations in terms of sample efficiency and computation time?

Key findings

• The small backup-based prioritized sweeping implementation achieved performance comparable to full value iteration with only one update cycle per time step, outperforming both classical implementations.
• With one update cycle per time step, the small backup method matched the performance of optimally tuned TD(0), despite not requiring step-size parameter tuning.
• The computation time per update cycle was lower for the small backup method, and the total computation time per update cycle was dominated by the O(P_re) term, indicating scalability.
• The Peng & Williams method performed worse than Moore & Atkeson’s method due to backups being proportional to transition probabilities (1/15), limiting their impact.
• The small backup method performed significantly more backups per update cycle—proportional to the number of predecessors—leading to faster propagation of value changes.
• The method demonstrated high sample efficiency, with the maximum standard deviation across 100 runs being only 0.1, except for Peng & Williams (1.0), indicating stable performance.