[Paper Review] Demonstration-Guided Reinforcement Learning with Learned Skills

Karl Pertsch, Youngwoon Lee+2 more|arXiv (Cornell University)|Jul 21, 2021

Reinforcement Learning in Robotics44 references9 citations

TL;DR

This paper proposes SkiLD, a demonstration-guided reinforcement learning method that leverages reusable skills learned from large offline, task-agnostic datasets to accelerate learning of new tasks. By guiding policy learning through demonstrated skills rather than primitive actions, SkiLD achieves significantly faster and more sample-efficient learning than prior methods on long-horizon navigation and robotic manipulation tasks.

ABSTRACT

Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors by leveraging both reward feedback and a set of target task demonstrations. Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator's exact muscle movements. Naturally, such learning will be slow, but often new behaviors are not completely unseen: they share subtasks with behaviors we have previously learned. In this work, we aim to exploit this shared subtask structure to increase the efficiency of demonstration-guided RL. We first learn a set of reusable skills from large offline datasets of prior experience collected across many tasks. We then propose Skill-based Learning with Demonstrations (SkiLD), an algorithm for demonstration-guided RL that efficiently leverages the provided demonstrations by following the demonstrated skills instead of the primitive actions, resulting in substantial performance improvements over prior demonstration-guided RL approaches. We validate the effectiveness of our approach on long-horizon maze navigation and complex robot manipulation tasks.

Motivation & Objective

• Address the inefficiency of existing demonstration-guided RL methods that treat each new task as independent and imitate primitive actions step-by-step.
• Leverage large-scale, task-agnostic offline datasets collected across diverse tasks to extract reusable, robust short-horizon skills.
• Improve sample efficiency in demonstration-guided RL by using a skill-based policy that chains previously learned skills instead of learning from scratch.
• Enable effective transfer to new tasks with limited demonstrations by aligning skill priors with task-relevant behavior distributions.
• Bridge the gap between imitation learning and skill-based RL by combining offline skill learning with task-specific demonstrations.

Proposed method

• Learn a set of reusable skills from a large offline dataset of diverse task experiences using a skill discovery algorithm (e.g., SPiRL-based approach).
• Construct a skill posterior from the offline data to guide exploration and policy optimization toward task-relevant skills.
• Design a reinforcement learning objective that combines a discriminator-based reward bonus with a learned skill posterior prior.
• Use demonstrations not to imitate primitive actions, but to guide the chaining of high-level skills via a skill-level policy.
• Optimize the policy using a combination of environment rewards, a GAIL-style discriminator bonus, and a skill posterior prior to stabilize training.
• Fine-tune the policy using a hybrid objective that encourages imitation of demonstrated states while leveraging the skill repertoire for efficient exploration.

Experimental results

Research questions

• RQ1 Can prior experience from a task-agnostic offline dataset significantly improve the sample efficiency of demonstration-guided RL on new tasks?
• RQ2 How does skill-based imitation compare to primitive-action imitation in terms of robustness and learning speed in complex, long-horizon tasks?
• RQ3 In what scenarios—particularly regarding data-task alignment—does combining demonstrations with offline skill priors provide the greatest benefit?
• RQ4 Can a learned skill posterior effectively guide policy learning when the offline dataset is misaligned with the target task?
• RQ5 How does the integration of demonstrations with skill priors compare to purely imitation-based or purely RL-based approaches in terms of convergence and final performance?

Key findings

• SkiLD achieves substantially faster learning and higher final performance than BC+RL and SPiRL on long-horizon maze navigation and robotic manipulation tasks.
• The method learns effective policies with significantly fewer demonstrations—demonstrating robustness to low-shot demonstration data by leveraging prior skill knowledge.
• Abllation studies show that removing the discriminator reward bonus slows convergence, while replacing the skill prior with the learned posterior leads to failure due to distributional misalignment.
• When the offline dataset is poorly aligned with the target task, SkiLD outperforms SPiRL by using demonstrations to correct the skill prior and guide the policy toward task-relevant behaviors.
• The method is most effective in scenarios where the offline data does not already contain task-relevant skills, highlighting its value in low-data, high-variability settings.
• In pure imitation settings without environment rewards, SkiLD still generalizes effectively, showing the robustness of the skill-based policy learning framework.