[Paper Review] Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning

Wen Sun, J. Andrew Bagnell+1 more|arXiv (Cornell University)|Feb 15, 2018

Reinforcement Learning in RoboticsComputer Science22 references38 citations

TL;DR

This paper proposes Truncated HORizon Policy Search (THOR), a method that combines imitation learning (IL) and reinforcement learning (RL) by shaping costs using a sub-optimal cost-to-go oracle. By truncating the planning horizon based on oracle accuracy, THOR enables faster, more sample-efficient learning than pure RL or IL baselines, achieving superior performance even with imperfect expert information.

ABSTRACT

In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a one-step greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal.

Motivation & Objective

• To bridge the gap between imitation learning (IL) and reinforcement learning (RL) by combining their strengths.
• To address the limitation of IL, where policies are capped at expert performance, even when the expert is sub-optimal.
• To develop a method that leverages expert cost-to-go estimates to accelerate learning while enabling improvement beyond the expert.
• To demonstrate that truncating the planning horizon based on oracle accuracy leads to better sample efficiency and performance.

Proposed method

• The method uses a cost-to-go oracle (ˆVe) to reshape the original MDP’s cost function via potential-based reward shaping.
• It introduces a truncated planning horizon of length k, where k is chosen based on the oracle's accuracy relative to the optimal value function V*.
• For discrete MDPs, the method mathematically guarantees a policy better than the expert, with a quantifiable performance gap.
• In practice, a gradient-based policy search algorithm is used, enabling application to continuous state and action spaces.
• The approach allows the use of complex function approximators and can be combined with existing RL algorithms like TRPO-GAE.
• The oracle ˆVe is typically pre-trained using TD learning on expert demonstrations, though online updates are also possible.

Experimental results

Research questions

• RQ1 How does the accuracy of a cost-to-go oracle affect the effective planning horizon in policy learning?
• RQ2 Can a truncated planning horizon improve sample efficiency and performance in RL when combined with IL?
• RQ3 Does cost shaping with a sub-optimal oracle enable policies to outperform both pure IL and RL baselines?
• RQ4 What is the optimal truncation length k for balancing bias, variance, and performance in the presence of imperfect oracles?

Key findings

• THOR with k > 1 significantly outperforms AGGREVATED (k = 1) in Acrobot and Mountain Car, especially in terms of mean + standard deviation, indicating better robustness and performance.
• In sparse-reward environments like Hopper and Swimmer, THOR with k ≈ 20–30% of the full horizon H achieved superior performance compared to TRPO-GAE and AGGREVATED.
• THOR reduced policy performance variance across runs, particularly in sparse-reward settings, due to shorter rollout lengths in policy gradient estimation.
• Even with an imperfect ˆVe, THOR with k = H (full horizon) still outperformed TRPO-GAE, confirming the benefit of reward shaping via oracle.
• The method achieved faster learning convergence than TRPO-GAE in all tested environments, including when the horizon was extended to H = 200 or H = 1000.