[Paper Review] A Versatile and Efficient Reinforcement Learning Framework for Autonomous Driving

Guan Wang, Haoyi Niu+4 more|arXiv (Cornell University)|Oct 22, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a versatile and efficient reinforcement learning framework for autonomous driving that decouples semantic representation learning from end-to-end policy learning. By using lightweight scene understanding for drivable space and lane detection, followed by distributed off-policy RL training, the framework achieves superior sim-to-real generalization, reduces training time to under 13 hours on a single GPU workstation, and demonstrates robust real-world performance across diverse scenarios including night driving and complex obstacle avoidance.

ABSTRACT

Heated debates continue over the best autonomous driving framework. The classic modular pipeline is widely adopted in the industry owing to its great interpretability and stability, whereas the fully end-to-end paradigm has demonstrated considerable simplicity and learnability along with the rise of deep learning. As a way of marrying the advantages of both approaches, learning a semantically meaningful representation and then use in the downstream driving policy learning tasks provides a viable and attractive solution. However, several key challenges remain to be addressed, including identifying the most effective representation, alleviating the sim-to-real generalization issue as well as balancing model training cost. In this study, we propose a versatile and efficient reinforcement learning framework and build a fully functional autonomous vehicle for real-world validation. Our framework shows great generalizability to various complicated real-world scenarios and superior training efficiency against the competing baselines.

Motivation & Objective

• Address the sim-to-real generalization gap in end-to-end autonomous driving by learning a semantic, low-dimensional representation from raw camera inputs.
• Overcome the limitations of modular pipelines (e.g., error compounding, rule-heavy design) and end-to-end systems (e.g., brittleness, poor interpretability) through a hybrid approach.
• Improve training efficiency and scalability of deep reinforcement learning for autonomous driving using a distributed, event-driven framework.
• Enable real-world deployment and validation of the framework on a physical autonomous vehicle across diverse and unseen scenarios.
• Achieve high autonomy and safety in complex environments, including night-time driving and dynamic obstacle avoidance, through a robust policy learned on semantic representations.

Proposed method

• Use a lightweight scene understanding model to extract semantic representations (drivable space and lane boundaries) from raw camera images, creating a canonical input space.
• Decouple representation learning from policy learning, enabling the downstream RL policy to operate on compact, semantically meaningful features.
• Implement a distributed, off-policy reinforcement learning framework with event-driven parallelization to accelerate training on a single workstation.
• Apply a flexible scheduling scheme that scales efficiently with the number of environments and actors, achieving consistent speedup.
• Integrate the learned RL policy with a PID controller for low-level control execution, ensuring stable and safe vehicle dynamics.
• Use a perception model that is 5× faster to train than CycleGAN while outperforming it in both speed and accuracy, reducing computational cost.

Experimental results

Research questions

• RQ1 Can a decoupled framework that learns semantic representations before policy learning improve sim-to-real generalization in autonomous driving?
• RQ2 How can distributed off-policy reinforcement learning be efficiently scaled on a single workstation to reduce training time without sacrificing performance?
• RQ3 To what extent can a representation-based RL framework generalize to unseen real-world scenarios, including complex road topologies and adverse lighting conditions?
• RQ4 How does the proposed framework compare in real-world autonomy and safety metrics against strong IL and RL baselines?
• RQ5 Can a lightweight, semantic-aware perception model outperform domain adaptation techniques like CycleGAN in both efficiency and performance for autonomous driving?

Key findings

• The proposed framework reduces total training time to 12.6 hours on a single workstation with 2 RTX3090 GPUs, a 85.6% speedup compared to the non-distributed baseline (87.2 hours).
• The framework achieves 98.3% success rate in obstacle avoidance and 96.7% in lane-keeping on straight roads, significantly outperforming the IL baseline in both autonomy and safety.
• In night-time conditions, the RL-based framework maintains high performance (92.5% success rate in obstacle avoidance), while the IL baseline shows a 12.1% drop in performance compared to day-time.
• The RL version exhibits higher steering angle standard deviation, indicating more aggressive yet effective obstacle bypassing, whereas the IL version frequently collides with obstacles and curbs.
• The perception model achieves better performance than CycleGAN with only 1/5 of the training time, demonstrating superior efficiency and effectiveness.
• Real-world deployment on a physical vehicle confirms the framework’s ability to generalize to unseen scenarios, including high-beam exposure, sidewalks, dense crowds, and garages, with consistent lane-following and dynamic obstacle avoidance.