[Paper Review] A Survey of Imitation Learning Methods, Environments and Metrics

Nathan Gavenski, Meneguzzi, Felipe+2 more|arXiv (Cornell University)|Apr 30, 2024

Human Motion and Animation4 citations

TL;DR

This survey introduces comprehensive, novel taxonomies for imitation learning methods, environments, and metrics—addressing a critical lack of standardization in evaluation. It systematically classifies approaches, evaluates their roles in learning, and highlights the need for consistent, behavior-focused metrics to improve comparability and real-world applicability in imitation learning research.

ABSTRACT

Imitation learning is an approach in which an agent learns how to execute a task by trying to mimic how one or more teachers perform it. This learning approach offers a compromise between the time it takes to learn a new task and the effort needed to collect teacher samples for the agent. It achieves this by balancing learning from the teacher, who has some information on how to perform the task, and deviating from their examples when necessary, such as states not present in the teacher samples. Consequently, the field of imitation learning has received much attention from researchers in recent years, resulting in many new methods and applications. However, with this increase in published work and past surveys focusing mainly on methodology, a lack of standardisation became more prominent in the field. This non-standardisation is evident in the use of environments, which appear in no more than two works, and evaluation processes, such as qualitative analysis, that have become rare in current literature. In this survey, we systematically review current imitation learning literature and present our findings by (i) classifying imitation learning techniques, environments and metrics by introducing novel taxonomies; (ii) reflecting on main problems from the literature; and (iii) presenting challenges and future directions for researchers.

Motivation & Objective

• Address the growing lack of standardization in imitation learning evaluation, particularly in environments and metrics.
• Systematically classify existing imitation learning methods beyond traditional on-policy/off-policy distinctions to reflect recent methodological trends.
• Introduce the first formal taxonomy of imitation learning environments based on their role in evaluation (validation, precision, sequential).
• Develop a novel taxonomy of metrics, categorizing them into behavior, domain, and model types with qualitative and quantitative subtypes.
• Highlight key challenges in evaluation, such as overreliance on environment-specific metrics and insufficient qualitative analysis, and propose future research directions.

Proposed method

• Adopt a snowballing literature review method using two foundational surveys (Hussein et al. 2017 and Zheng et al. 2021) as starting points.
• Propose a new taxonomy for imitation learning methods that emphasizes emerging trends, complementing existing on-policy/off-policy classifications.
• Introduce a three-tiered taxonomy for environments based on their functional role: validation (for policy evaluation), precision (for high-fidelity behavior matching), and sequential (for long-horizon tasks).
• Develop a three-category metric taxonomy: behavior (reward-based and distance-based), domain (task-level performance), and model (policy and representation quality).
• Use formal definitions of imitation learning components (e.g., MDPs, teacher demonstrations, reward functions) to ensure mathematical rigor in methodological classification.
• Conduct a systematic review of 240+ papers to identify patterns in environments, metrics, and methods, focusing on consistency and reproducibility in evaluation.

(a) CartPole [ 77 ]

Experimental results

Research questions

• RQ1 How can imitation learning methods be systematically classified beyond traditional on-policy and off-policy distinctions?
• RQ2 What roles do different types of environments play in evaluating imitation learning agents, and how can they be categorized meaningfully?
• RQ3 What are the key limitations in current evaluation practices, and how do existing metrics fail to capture behavioral fidelity or generalization?
• RQ4 To what extent do current metrics support meaningful comparison across imitation learning methods, and can they be unified or aggregated?
• RQ5 What are the most pressing challenges in evaluation standardization, and how can future research improve consistency and human-centric performance assessment?

Key findings

• The field lacks standardization in evaluation, with environments used in no more than two works and qualitative analysis rarely reported, hindering method comparison.
• The proposed taxonomy for environments categorizes them into validation, precision, and sequential roles, enabling researchers to better understand and select evaluation settings.
• The metric taxonomy groups evaluation into behavior, domain, and model categories, with behavior metrics further split into reward-based and distance-based types, offering diverse performance perspectives.
• Despite progress, imitation learning methods still rely heavily on environment-specific metrics, often neglecting behavioral fidelity and generalization—key for real-world deployment.
• The survey identifies a critical gap in evaluating agent behavior beyond reward or success rates, advocating for more human-centric metrics inspired by reinforcement learning from human feedback.
• The authors conclude that consistent use of standardized environments and metrics is essential for reliable benchmarking and future advancement in imitation learning.

(a) MuJoCo [ 82 ]