[Paper Review] Distilling Policy Distillation

Wojciech Marian Czarnecki, Razvan Pascanu+4 more|arXiv (Cornell University)|Feb 6, 2019

Reinforcement Learning in Robotics33 references39 citations

TL;DR

The paper surveys the policy distillation landscape in reinforcement learning, analyzes when on-policy student-driven distillation yields convergence, and proposes the best-practice of expected entropy regularised distillation, plus methods to leverage teacher critics.

ABSTRACT

The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the widespread use and conceptual simplicity of distillation, many different formulations are used in practice, and the subtle variations between them can often drastically change the performance and the resulting objective that is being optimised. In this work, we rigorously explore the entire landscape of policy distillation, comparing the motivations and strengths of each variant through theoretical and empirical analysis. Our results point to three distillation techniques, that are preferred depending on specifics of the task. Specifically a newly proposed expected entropy regularised distillation allows for quicker learning in a wide range of situations, while still guaranteeing convergence.

Motivation & Objective

• Clarify the spectrum of policy distillation formulations used in RL and their motivations.
• Characterize when on-policy, student-driven distillation converges and when it can oscillate or fail.
• Investigate how actor-critic settings and teacher value functions can aid distillation.
• Propose robust distillation variants and a practical decision guide for choosing methods.
• Empirically compare distillation approaches in thousands of random MDPs to derive best practices.

Proposed method

• Systematically compare distillation variants (teacher distill, on-policy distill, on-policy distill+R, entropy regularised, N-distill, exp. entropy regularised, Teacher V reward) both theoretically and empirically.
• Prove that on-policy distillation with trajectories from the student does not form a gradient vector field in general and can oscillate when rewards are present, and show how to recover gradient-field properties.
• Introduce and validate the expected entropy regularised distillation as a gradient-friendly, low-variance, and faithful replication method.
• Explore actor-critic scenarios where the teacher’s value function is available and show how V has a gating effect to improve or bootstrap learning.
• Use thousands of random MDPs to compare control-policy choices (teacher-driven, student-driven, uniform) and update rules, identifying practical guidelines.

Experimental results

Research questions

• RQ1 Under what conditions does on-policy, student-driven distillation converge or oscillate?
• RQ2 Can a gradient-vector-field consistent update be achieved for distillation when rewards are present?
• RQ3 How can teacher value functions be incorporated to improve distillation in actor-critic settings?
• RQ4 Which distillation variant provides reliable convergence and fast learning across diverse MDPs?
• RQ5 What practical guidelines emerge for selecting distillation methods based on task characteristics?

Key findings

• On-policy distillation using student-generated trajectories generally does not form a gradient vector field and can fail to converge when rewards are involved; a gradient-consistent correction is possible.
• Student-driven distillation leads to faster and broader replication of teacher behaviour across the state distribution relevant to the student, outperforming teacher-driven distillation in empirical tests.
• The proposed expected entropy regularised distillation is the most reliable variant, combining gradient-field validity, low variance, and effective trajectory-level cloning.
• Using the teacher’s critic to gate or bootstrap learning can recover or improve performance when teachers are imperfect or noisy.
• In actor-critic setups, leveraging teacher value functions can ensure the student’s performance is at least as good as the teacher under certain conditions; intrinsic rewards from the teacher’s critic shape learning dynamics.