[Paper Review] Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning

Peide Cai, Hengli Wang+3 more|arXiv (Cornell University)|Jul 18, 2021

Reinforcement Learning in Robotics28 references6 citations

TL;DR

This paper proposes Deep Imitative Reinforcement Learning (DIRL), a unified framework that combines imitation learning (IL) and model-based reinforcement learning (RL) to train end-to-end visual control policies for autonomous car racing. By leveraging an uncertainty-aware world model (Reveries-net) to simulate imagined interactions, DIRL achieves superior sample efficiency and performance over pure IL or RL baselines, outperforming them in both simulation and real-world 1/20-scale RC-car racing with reduced need for expert interventions.

ABSTRACT

Autonomous car racing is a challenging task in the robotic control area. Traditional modular methods require accurate mapping, localization and planning, which makes them computationally inefficient and sensitive to environmental changes. Recently, deep-learning-based end-to-end systems have shown promising results for autonomous driving/racing. However, they are commonly implemented by supervised imitation learning (IL), which suffers from the distribution mismatch problem, or by reinforcement learning (RL), which requires a huge amount of risky interaction data. In this work, we present a general deep imitative reinforcement learning approach (DIRL), which successfully achieves agile autonomous racing using visual inputs. The driving knowledge is acquired from both IL and model-based RL, where the agent can learn from human teachers as well as perform self-improvement by safely interacting with an offline world model. We validate our algorithm both in a high-fidelity driving simulation and on a real-world 1/20-scale RC-car with limited onboard computation. The evaluation results demonstrate that our method outperforms previous IL and RL methods in terms of sample efficiency and task performance. Demonstration videos are available at https://caipeide.github.io/autorace-dirl/

Motivation & Objective

• To address the limitations of traditional modular and end-to-end deep learning methods in autonomous car racing, particularly the distribution shift in IL and sample inefficiency in RL.
• To develop a unified learning framework that combines the data efficiency of IL with the self-improvement capability of model-based RL.
• To enable safe and efficient policy training using an offline world model, minimizing real-world interactions and expert interventions.
• To validate the method in both high-fidelity simulation and real-world 1/20-scale RC-car racing with limited onboard computation.

Proposed method

• Proposes Reveries-net, a recurrent neural network with evidential uncertainty estimation to learn an action-conditioned, probabilistic world model from visual inputs.
• Uses IL to initialize a visual control policy with expert demonstration data, providing a strong starting policy.
• Applies model-based RL to refine the policy by simulating future trajectories in the learned world model, enabling safe self-improvement without real-world interaction.
• Integrates uncertainty estimation as a regularizing signal during policy training, improving robustness and safety.
• Employs a closed-loop replanning strategy where the policy re-evaluates actions at every time step based on real-time observations and predicted outcomes.
• Trains the policy exclusively in the offline world model, reducing reliance on real-world rollouts and enabling data-efficient learning.

Experimental results

Research questions

• RQ1 Can a unified framework combining IL and model-based RL achieve better sample efficiency and task performance than standalone IL or RL in autonomous racing?
• RQ2 How does uncertainty-aware world modeling improve policy safety and robustness during offline training?
• RQ3 To what extent can a policy trained in a simulated world model generalize to real-world RC-car racing with minimal real-world fine-tuning?
• RQ4 Does the integration of human demonstrations and imagined interactions lead to faster convergence and reduced need for expert interventions?

Key findings

• DIRL achieves 1.8 to 8.4 times higher data efficiency than prior methods in simulation, significantly reducing the number of expert demonstrations and real-world rollouts required.
• DIRL(3 iter) requires only 1.33 and 4.33 interventions in easy and hard tasks, respectively, outperforming previous versions and reducing reliance on expert correction.
• Removing uncertainty estimation (DIRL(-unc)) leads to a dramatic performance drop, with 154.5 seconds and 32.67 interventions in the hard task, indicating uncertainty is critical for policy safety.
• The policy trained in Reveries-net generalizes well to real-world RC-car racing, enabling agile, safe navigation through complex obstacles and sharp turns.
• Visualizations show Reveries-net accurately predicts future states, with high uncertainty near collision points, demonstrating its reliability in risk estimation.
• Despite strong performance, DIRL still lags behind human racers by approximately 10 seconds in the hard task, indicating room for improvement in policy architecture and world model fidelity.