[Paper Review] ApolloRL: a Reinforcement Learning Platform for Autonomous Driving

Fei Gao, Peng Geng+8 more|arXiv (Cornell University)|Jan 29, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

ApolloRL is an open, end-to-end reinforcement learning platform for autonomous driving that integrates real-world driving data, simulation, training, and evaluation in a scalable, distributed architecture. It enables researchers to train and benchmark RL agents on 300 hours of real-world driving scenarios using SOTA algorithms like PPO and SAC, achieving stable, reproducible performance improvements in complex traffic environments.

ABSTRACT

We introduce ApolloRL, an open platform for research in reinforcement learning for autonomous driving. The platform provides a complete closed-loop pipeline with training, simulation, and evaluation components. It comes with 300 hours of real-world data in driving scenarios and popular baselines such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents. We elaborate in this paper on the architecture and the environment defined in the platform. In addition, we discuss the performance of the baseline agents in the ApolloRL environment.

Motivation & Objective

• To address the challenges of autonomous driving decision-making under imperfect information, multiple objectives, complex action spaces, and delayed rewards.
• To provide a unified, scalable, and extensible platform for reinforcement learning research in real-world autonomous driving scenarios.
• To bridge the gap between simulation and real-world deployment by leveraging large-scale, high-fidelity real-world driving data.
• To support rapid iteration and evaluation of RL algorithms through integrated training, simulation, and evaluation pipelines.
• To inspire and accelerate research in deep reinforcement learning for autonomous driving through open access to data, tools, and community collaboration.

Proposed method

• The platform uses a closed-loop pipeline integrating perception, planning, control, and simulation modules to enable end-to-end RL training.
• It leverages 300 hours of real-world driving data (50k for training, 10k for testing) to generate realistic, diverse traffic scenarios in simulation.
• The environment supports customizable observations, actions, and shaped rewards, enabling flexible RL algorithm implementation.
• Distributed simulation and distributed training components are used to scale up training and evaluation across multiple agents and scenarios.
• Baseline agents are pre-trained with supervised learning from expert driver logs and then refined via reinforcement learning in the virtual environment.
• The platform supports popular continuous control RL algorithms, including Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC).

Experimental results

Research questions

• RQ1 Can a unified, open platform significantly accelerate research in deep reinforcement learning for autonomous driving?
• RQ2 How effectively can RL agents trained on simulated, real-world-derived scenarios generalize to complex, long-horizon driving tasks?
• RQ3 To what extent can PPO and SAC agents improve performance on real-world driving scenarios compared to baseline approaches?
• RQ4 Can the platform support scalable, reproducible training and evaluation of RL agents across diverse, high-fidelity traffic environments?
• RQ5 What is the potential of combining supervised pre-training with reinforcement fine-tuning in autonomous driving decision-making?

Key findings

• The ApolloRL platform successfully enables stable and reproducible training of RL agents, as evidenced by steadily improving accumulated reward curves over training steps.
• Baseline agents trained with PPO and SAC achieve measurable performance gains in complex driving scenarios, with some outperforming the baseline in the on-road autonomous driving system.
• The platform supports the training of agents on dozens of diverse Autonomous Driving Scenarios (ADS), with a large proportion achieving the training objectives within 20–30 seconds per scenario.
• The integration of real-world data into simulation enables high-fidelity scenario generation that supports realistic agent training and evaluation.
• The platform’s distributed architecture enables efficient scaling of simulation and training, supporting future expansion to thousands of scenarios.
• The platform is extensible and supports user-defined algorithms, hyperparameters, models, and exploration strategies, enabling broad research applicability.