[Paper Review] Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies

Yu Huang, Yue Chen|arXiv (Cornell University)|Jun 10, 2020

Autonomous Vehicle Technology and Safety234 references34 citations

TL;DR

This survey reviews deep learning methods across autonomous driving, focusing on perception, mapping/localization, prediction, planning/control, simulation, V2X, and safety, with emphasis on 2D/3D object detection, depth estimation, and multi-sensor fusion.

ABSTRACT

Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Almost at the same time, deep learning has made breakthrough by several pioneers, three of them (also called fathers of deep learning), Hinton, Bengio and LeCun, won ACM Turin Award in 2019. This is a survey of autonomous driving technologies with deep learning methods. We investigate the major fields of self-driving systems, such as perception, mapping and localization, prediction, planning and control, simulation, V2X and safety etc. Due to the limited space, we focus the analysis on several key areas, i.e. 2D and 3D object detection in perception, depth estimation from cameras, multiple sensor fusion on the data, feature and task level respectively, behavior modelling and prediction of vehicle driving and pedestrian trajectories.

Motivation & Objective

• Provide a comprehensive overview of deep learning-based autonomous driving technologies.
• Analyze major system components: perception, mapping/localization, prediction, planning/control, simulation, V2X, and safety.
• Highlight key DL techniques in 2D/3D object detection, depth estimation from cameras, and multi-sensor fusion (feature- and task-level).
• Discuss behavior modeling and trajectory prediction for vehicles and pedestrians.
• Identify challenges and potential future directions in DL-powered autonomous driving.

Proposed method

• Conduct a systematic literature review of deep learning methods applicable to autonomous driving.
• Categorize the discussion around core subsystems and focal topics in the field.
• Evaluate and synthesize advances in perception, including 2D/3D object detection and depth estimation from cameras.
• Examine multi-sensor fusion approaches at both feature-level and task-level.
• Summarize behavior modeling and trajectory prediction for agents in driving scenarios.

Experimental results

Research questions

• RQ1 What deep learning approaches enable robust 2D and 3D object detection for autonomous driving?
• RQ2 How can depth estimation from cameras be effectively achieved for driving scenarios?
• RQ3 How are multi-sensor fusion techniques applied to improve perception and localization?
• RQ4 How are vehicle and pedestrian trajectories modeled and predicted using DL?
• RQ5 What are the current challenges and future directions for deep learning in autonomous driving?

Key findings

• Deep learning is central to perception, mapping/localization, prediction, planning/control, simulation, V2X, and safety in autonomous driving.
• In perception, 2D/3D object detection and depth estimation from cameras are key focus areas.
• Multi-sensor fusion at both feature- and task-level is crucial for robust perception and localization.
• Behavior modeling and trajectory prediction for vehicles and pedestrians are essential components.
• The survey identifies ongoing challenges and outlines potential directions for future DL developments in autonomous driving.