[Paper Review] AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers

Andrey Kurenkov, Ajay Mandlekar+3 more|arXiv (Cornell University)|Sep 9, 2019

Reinforcement Learning in Robotics38 references17 citations

TL;DR

AC-Teach proposes a Bayesian actor-critic framework that leverages an ensemble of suboptimal, partial, or contradictory teachers to improve sample efficiency in deep reinforcement learning. By using probabilistic value estimation and Thompson sampling over teacher-proposed actions, it enables stable, adaptive exploration and outperforms baselines in both convergence speed and asymptotic performance across robotic manipulation tasks, even with low-quality or incomplete teacher sets.

ABSTRACT

The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency. Thus, improving over random exploration is crucial to solve long-horizon tasks with sparse rewards. We propose to leverage an ensemble of partial solutions as teachers that guide the agent's exploration with action suggestions throughout training. While the setup of learning with teachers has been previously studied, our proposed approach - Actor-Critic with Teacher Ensembles (AC-Teach) - is the first to work with an ensemble of suboptimal teachers that may solve only part of the problem or contradict other each other, forming a unified algorithmic solution that is compatible with a broad range of teacher ensembles. AC-Teach leverages a probabilistic representation of the expected outcome of the teachers' and student's actions to direct exploration, reduce dithering, and adapt to the dynamically changing quality of the learner. We evaluate a variant of AC-Teach that guides the learning of a Bayesian DDPG agent on three tasks - path following, robotic pick and place, and robotic cube sweeping using a hook - and show that it improves largely on sampling efficiency over a set of baselines, both for our target scenario of unconstrained suboptimal teachers and for easier setups with optimal or single teachers. Additional results and videos at https://sites.google.com/view/acteach/home.

Motivation & Objective

• To improve sample efficiency in deep reinforcement learning for long-horizon robotic tasks with sparse rewards.
• To enable effective policy learning when teachers are suboptimal, partial, or contradictory, rather than requiring optimal or complete expert demonstrations.
• To develop a unified, scalable framework that integrates heterogeneous teacher advice into the behavioral policy without explicit conditioning on individual teachers.
• To maintain stability and performance when training from off-policy experience collected via a mix of agent and teacher actions.
• To enable the agent to learn behaviors not covered by any teacher, even when the teacher set is insufficient or noisy.

Proposed method

• AC-Teach extends the actor-critic architecture with a behavioral policy that selects actions based on Thompson sampling from a probabilistic posterior over action values predicted by a Bayesian critic.
• The Bayesian critic maintains a posterior distribution over Q-values, enabling uncertainty-aware selection between agent actions and teacher-proposed actions.
• Teacher advice is integrated as alternative action proposals without explicit conditioning, allowing the method to scale to large or dynamic teacher sets.
• A commitment mechanism allows the agent to follow a single teacher policy for extended sequences, reducing dithering and improving policy stability.
• The method uses off-policy experience replay, enabling training from heterogeneous data generated by the agent and multiple teachers.
• The framework is compatible with any off-policy RL algorithm, demonstrated here with Bayesian DDPG (BDDPG).

Experimental results

Research questions

• RQ1 Can AC-Teach significantly improve sample efficiency when using an ensemble of suboptimal, partial, or contradictory teachers?
• RQ2 How does AC-Teach perform when the teacher set is insufficient, requiring the agent to learn parts of the task from scratch?
• RQ3 How robust is AC-Teach to noisy or low-quality teachers, including random action providers?
• RQ4 Does AC-Teach outperform baselines in both convergence speed and final performance across diverse teacher configurations?
• RQ5 What is the contribution of each component (uncertainty estimation, commitment, ensemble integration) to overall performance?

Key findings

• AC-Teach significantly outperforms all baselines in sample efficiency and asymptotic performance on the pick-and-place, path-following, and hook-sweeping tasks.
• Even with a single noisy teacher, AC-Teach learns effectively and approaches the performance of a noisy full teacher, while all baselines fail to learn.
• When using multiple partial teachers (e.g., separate pick and place controllers), AC-Teach outperforms a hand-coded, near-optimal teacher with noise.
• The method remains robust when adding random teachers; performance degrades slowly, but AC-Teach consistently outperforms all baselines.
• Ablation studies confirm that all four components—Bayesian critic, Thompson sampling, commitment mechanism, and ensemble integration—are essential for strong performance.
• The addition of suboptimal or partial teachers improves performance over optimal or single teachers, suggesting that diversity in teacher advice mitigates extrapolation error.