[Paper Review] SAFARI: Safe and Active Robot Imitation Learning with Imagination

Norman Di Palo, Edward Johns|arXiv (Cornell University)|Nov 18, 2020

Robot Manipulation and Learning42 references4 citations

TL;DR

SAFARI proposes a safe and active imitation learning framework that combines behavioral cloning with model-based planning and uncertainty estimation to minimize distribution shift and predict failures. By actively querying for demonstrations in out-of-distribution states and using future state prediction to detect anomalies, SAFARI improves generalization and safety in robotic manipulation tasks.

ABSTRACT

One of the main issues in Imitation Learning is the erroneous behavior of an agent when facing out-of-distribution situations, not covered by the set of demonstrations given by the expert. In this work, we tackle this problem by introducing a novel active learning and control algorithm, SAFARI. During training, it allows an agent to request further human demonstrations when these out-of-distribution situations are met. At deployment, it combines model-free acting using behavioural cloning with model-based planning to reduce state-distribution shift, using future state reconstruction as a test for state familiarity. We empirically demonstrate how this method increases the performance on a set of manipulation tasks with respect to passive Imitation Learning, by gathering more informative demonstrations and by minimizing state-distribution shift at test time. We also show how this method enables the agent to autonomously predict failure rapidly and safely.

Motivation & Objective

• To address the problem of erroneous behavior in imitation learning when agents encounter out-of-distribution (OOD) states not covered in expert demonstrations.
• To reduce state-distribution shift during deployment by combining model-free policy acting with online planning that minimizes divergence from demonstrated states.
• To enable autonomous, safe failure prediction at test time without human supervision, improving robot safety.
• To enhance data efficiency during training by actively selecting the most uncertain, and thus most informative, states for expert demonstration.

Proposed method

• Trains a policy network via behavioral cloning on expert demonstrations to imitate expert actions.
• Learns a dynamics model (world model) to predict future states from current states and actions, enabling online planning.
• Employs an epistemic uncertainty network to estimate uncertainty in predictions, identifying OOD states via future state reconstruction.
• Uses uncertainty estimates to trigger active learning during training by querying the expert for demonstrations in high-uncertainty states.
• At test time, combines the policy’s action with a planning action that minimizes uncertainty and pulls the state back toward the training distribution.
• Employs a convolutional autoencoder to compress RGB observations into a 16-dimensional latent vector, which is concatenated with robot state features for input.

Experimental results

Research questions

• RQ1 Can active learning based on uncertainty estimation reduce the number of required expert demonstrations while improving policy generalization?
• RQ2 To what extent can model-based planning reduce state-distribution shift during deployment in imitation learning?
• RQ3 Can future state prediction via a learned dynamics model enable faster and more reliable failure detection compared to reactive methods?
• RQ4 How does combining model-free policy acting with uncertainty-aware planning improve performance and safety on real-world manipulation tasks?

Key findings

• The active learning approach with γ = 0.8 achieved superior performance compared to passive imitation learning, demonstrating that actively selected demonstrations are more informative.
• In the 1-Cube Pick and Place task, SAFARI achieved 8.8 ± 2.35 successful test instances out of 50, significantly outperforming behavior cloning (7.6 ± 2.35).
• For the 2-Cubes Stack task, SAFARI achieved 5.9 ± 4.0 successes, compared to 3.4 ± 2.6 for behavior cloning, showing improved robustness on complex tasks.
• The hybrid policy combining model-free acting and online planning reduced distribution shift and improved test-time performance across all evaluated tasks.
• Failure prediction using future state imagination achieved high F1 scores (stable from 0 to 10 future steps) while reducing the number of steps required to detect failure, enhancing safety.
• The method successfully detected dangerous states autonomously at test time, enabling safe execution without human intervention.