[Paper Review] Deep Imitative Models for Flexible Inference, Planning, and Control

Nicholas Rhinehart, Rowan McAllister+1 more|arXiv (Cornell University)|Oct 15, 2018

Reinforcement Learning in Robotics39 references55 citations

TL;DR

Imitative Models learn a probabilistic model of expert trajectories and use test-time goal likelihoods to plan expert-like trajectories to achieve flexible goals, combining imitation learning with planning without reward engineering.

ABSTRACT

Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.

Motivation & Objective

• Motivate the need for flexible goal-directed control beyond traditional imitation learning and reward-based planning.
• Propose a probabilistic imitative model that forecasts expert trajectories conditioned on scene observations.
• Develop a planning objective that combines the imitation prior with a test-time goal likelihood to produce expert-like plans.
• Showcase robustness and flexibility across various goal specifications and test-time conditions in autonomous driving.

Proposed method

• Train an imitative model q(S1:T | φ) to forecast expert trajectories from offline demonstrations.
• Use a probabilistic trajectory density (R2P2-based autoregressive flow) to model expert-like behavior and enable gradient-based planning.
• Formulate a maximum a posteriori planning objective: s* = argmax_s log q(s|φ) + log p(G|s,φ) - log p(G|φ).
• Construct diverse goal likelihoods p(G|s,φ): Final-State Indicator (region/line/point constraints), Gaussian Final-State (single or multiple future states), Gaussian State Sequence, and Gaussian Final-State Mixture with optional test-time costs.
• Instantiate the model for autonomous driving in CARLA using route waypoints as goals, LIDAR/camera inputs, and a PID low-level controller.
• Employ an attention-augmented neural architecture (mθ, σθ) to parameterize q(S|φ) with inputs including past states, perception χ, traffic signal λ, and latent Z.

Experimental results

Research questions

• RQ1 Can offline-trained imitative models generate interpretable, expert-like multi-step plans without reward engineering?
• RQ2 How flexible is the approach to achieve goals not seen during training (e.g., region-based goals, pothole avoidance) at test time?
• RQ3 How robust is goal specification to noise, mis-specified goals, or decoy waypoints?
• RQ4 Do the proposed methods achieve state-of-the-art performance in dynamic autonomous driving benchmarks (CARLA) using standard sensor inputs?
• RQ5 What is the impact of incorporating test-time costs or various goal likelihood forms on planning quality?

Key findings

• The method yields interpretable, expert-like multi-step plans without reward engineering and outperforms six imitation-learning approaches and a planning-based baseline in CARLA.
• Imitative planning with various goal likelihoods achieves strong performance under both training and test conditions, including dynamic scenes.
• The approach demonstrates robustness to noisy or mis-specified goals, including goals on the wrong side of the road and decoy waypoints.
• In experiments, the method attains state-of-the-art or competitive CARLA performance using common autonomous-driving inputs (waypoints and LIDAR).
• Test-time costs (e.g., pothole avoidance) can be incorporated to produce safe, goal-directed behavior not demonstrated in training.
• The framework supports flexibility to new tasks without retraining by leveraging the imitation prior and goal likelihoods.