[Paper Review] Curriculum Design for Teaching via Demonstrations: Theory and Applications

Gaurav Yengera, Rati Devidze+2 more|arXiv (Cornell University)|Jun 8, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a unified, interactive curriculum strategy for teaching via demonstrations in imitation learning, ranking demonstrations by difficulty based on the ratio of likelihood under the teacher's optimal policy and the learner's current policy. It achieves linear convergence for MaxEnt-IRL and CrossEnt-BC learners without requiring access to the learner’s internal dynamics, and extends to teacherless settings using task-specific difficulty scores, outperforming prior methods in synthetic navigation and car driving environments.

ABSTRACT

We consider the problem of teaching via demonstrations in sequential decision-making settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher's optimal policy and the learner's current policy. Compared to the state of the art, our strategy doesn't require access to the learner's internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore, we adapt our curriculum strategy to the setting where no teacher agent is present using task-specific difficulty scores. Experiments on a synthetic car driving environment and navigation-based environments demonstrate the effectiveness of our curriculum strategy.

Motivation & Objective

• To address the challenge of reducing the number of demonstrations needed for efficient imitation learning by designing a personalized curriculum.
• To develop a curriculum strategy that is agnostic to the learner’s internal dynamics, enabling broad applicability across different learner models.
• To extend the curriculum approach to settings without a teacher agent by using task-specific difficulty scores.
• To provide theoretical convergence guarantees for MaxEnt-IRL and CrossEnt-BC learners under mild technical conditions.
• To empirically validate the effectiveness of the curriculum strategy in synthetic navigation and car driving environments.

Proposed method

• The curriculum strategy ranks demonstrations based on the ratio of their likelihood under the teacher’s optimal policy to their likelihood under the learner’s current policy, serving as a difficulty score.
• For MaxEnt-IRL, the method proves linear convergence under mild technical conditions, without requiring knowledge of the learner’s update rule.
• For CrossEnt-BC, the same difficulty-based ranking is applied, enabling a unified curriculum strategy across both learner models.
• A scheduling mechanism is used to gradually increase the number of demonstrations provided per training epoch, starting with the most preferred (difficult) ones and eventually including all.
• In the absence of a teacher, task-specific difficulty scores are derived from structural properties of the environment, such as number of goals or greedy gap, to guide curriculum generation.
• The approach is evaluated on a synthetic car driving environment and two navigation-based environments, with training using neural networks and curriculum-scheduled demonstration batches.

Experimental results

Research questions

• RQ1 Can a unified curriculum strategy be designed for both MaxEnt-IRL and CrossEnt-BC learners using only policy-based difficulty scores?
• RQ2 Does the proposed curriculum achieve linear convergence for MaxEnt-IRL without access to the learner’s internal dynamics?
• RQ3 Can the curriculum strategy be adapted to settings without a teacher agent using only task-specific features?
• RQ4 How does the curriculum strategy compare to prior batch and interactive teaching methods in terms of convergence speed and sample efficiency?
• RQ5 What structural patterns emerge in the curriculum selection, and do they align with intuitive learning progressions in navigation tasks?

Key findings

• The proposed curriculum strategy achieves linear convergence for MaxEnt-IRL learners under mild technical conditions, even without access to the learner’s update rule.
• In the shortest path navigation environment, the curriculum prioritizes tasks with fewer goals and more obstacles early on, teaching obstacle avoidance before optimal path selection.
• In the TSP navigation environment, the curriculum begins with tasks that have more goals but lower greedy gap, gradually introducing more complex planning tasks.
• The curriculum strategy significantly accelerates learning convergence compared to uniform and random demonstration scheduling in both synthetic environments.
• The method generalizes effectively to teacherless settings by using task-specific difficulty scores derived from environmental features such as number of goals and greedy gap.
• Empirical results show that the curriculum strategy outperforms state-of-the-art methods in terms of sample efficiency and convergence speed, particularly in complex navigation tasks.