[Paper Review] Importance Resampling for Off-policy Prediction

Matthew Schlegel, Wesley Chung+3 more|arXiv (Cornell University)|Jun 11, 2019

Reinforcement Learning in Robotics35 references4 citations

TL;DR

This paper introduces Importance Resampling (IR), a novel off-policy prediction method that replaces importance sampling reweighting with resampling from a replay buffer using IS ratios as sampling weights. IR achieves lower update variance and faster convergence than IS, WIS, and V-trace in both tabular and deep RL settings, particularly reducing learning rate sensitivity and improving sample efficiency.

ABSTRACT

Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the weights for the value function. In this work, we explore a resampling strategy as an alternative to reweighting. We propose Importance Resampling (IR) for off-policy prediction, which resamples experience from a replay buffer and applies standard on-policy updates. The approach avoids using importance sampling ratios in the update, instead correcting the distribution before the update. We characterize the bias and consistency of IR, particularly compared to Weighted IS (WIS). We demonstrate in several microworlds that IR has improved sample efficiency and lower variance updates, as compared to IS and several variance-reduced IS strategies, including variants of WIS and V-trace which clips IS ratios. We also provide a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.

Motivation & Objective

• Address the high variance problem in off-policy value function learning caused by large importance sampling (IS) ratios.
• Explore resampling as an alternative to reweighting in off-policy prediction, avoiding direct multiplication of updates by extreme IS ratios.
• Demonstrate that resampling before learning can stabilize training and improve convergence speed compared to IS-based methods.
• Evaluate IR in both tabular and deep RL settings, including image-based control tasks, to assess generalization and robustness.
• Compare IR to IS, WIS, and V-trace in terms of variance, learning rate sensitivity, and convergence speed across multiple environments.

Proposed method

• Resample transitions from a replay buffer using IS ratios as sampling probabilities: P(s,a) ∝ π(a|s)/μ(a|s).
• Apply standard on-policy updates (e.g., TD(0)) to the resampled transitions, avoiding multiplication by IS ratios in the update rule.
• Use a weighted bootstrap approach to correct the distribution of experience before learning, ensuring unbiased estimation.
• Introduce a bias-corrected variant of IR to maintain consistency while reducing bias from resampling.
• Implement IR with mini-batch updates and buffer management to support online learning and scalability.
• Compare IR to IS, WIS (with buffer and minibatch variants), and V-trace (with varying clipping thresholds) in identical experimental setups.

Experimental results

Research questions

• RQ1 Does resampling experience using IS ratios as weights reduce update variance compared to standard IS and WIS in off-policy prediction?
• RQ2 How does IR compare to IS and V-trace in terms of learning rate sensitivity and convergence speed in tabular and deep RL environments?
• RQ3 Can IR achieve faster convergence in terms of number of weight updates while maintaining consistency and low bias?
• RQ4 Does IR improve performance on high-variance, high-ratio settings such as image-based control tasks with large IS ratios?
• RQ5 How does the variance of IR updates compare to IS and WIS during early training phases?

Key findings

• IR achieves significantly lower update variance than IS and WIS in early training, particularly in the Random Walk Markov Chain environment.
• IR shows the lowest learning rate sensitivity among all methods, matching the performance stability of on-policy learning.
• In the Four Rooms environment, IR converges faster than IS and WIS, with lower variance and improved sample efficiency.
• In the TORCs racing car simulator with image observations, IR outperforms IS and V-trace, especially at higher learning rates, indicating reduced variance.
• IR maintains consistency and has bias comparable to WIS, with a bias-corrected variant achieving unbiased estimation.
• The variance of IR updates is consistently lower than IS across training iterations, correlating with reduced learning rate sensitivity and faster convergence.