[Paper Review] YOLOP: You Only Look Once for Panoptic Driving Perception

Dong Wu, Manwen Liao+5 more|arXiv (Cornell University)|Aug 25, 2021

Advanced Neural Network Applications35 references4 citations

TL;DR

YOLOP proposes a real-time, end-to-end multi-task network for panoptic driving perception, jointly performing object detection, drivable area segmentation, and lane detection using a single encoder and three task-specific decoders. It achieves state-of-the-art performance on the BDD100K dataset and runs at 23 FPS on the Jetson TX2 embedded device, making it the first such system to deliver real-time inference on edge hardware while maintaining high accuracy.

ABSTRACT

A panoptic driving perception system is an essential part of autonomous driving. A high-precision and real-time perception system can assist the vehicle in making the reasonable decision while driving. We present a panoptic driving perception network (YOLOP) to perform traffic object detection, drivable area segmentation and lane detection simultaneously. It is composed of one encoder for feature extraction and three decoders to handle the specific tasks. Our model performs extremely well on the challenging BDD100K dataset, achieving state-of-the-art on all three tasks in terms of accuracy and speed. Besides, we verify the effectiveness of our multi-task learning model for joint training via ablative studies. To our best knowledge, this is the first work that can process these three visual perception tasks simultaneously in real-time on an embedded device Jetson TX2(23 FPS) and maintain excellent accuracy. To facilitate further research, the source codes and pre-trained models are released at https://github.com/hustvl/YOLOP.

Motivation & Objective

• To develop a unified, real-time perception system for autonomous driving that jointly handles object detection, drivable area segmentation, and lane detection.
• To address the challenge of deploying high-precision, low-latency perception on resource-constrained embedded devices like Jetson TX2.
• To validate the effectiveness of end-to-end multi-task learning without requiring complex alternating optimization.
• To investigate whether grid-based detection heads are more compatible with semantic segmentation tasks than region-based approaches.

Proposed method

• A lightweight CNN encoder extracts multi-scale features from a single input image.
• Three decoders are used: one for object detection (based on YOLOv4), one for drivable area segmentation, and one for lane line segmentation.
• All tasks are trained end-to-end using a shared backbone, enabling feature sharing and joint optimization.
• The model leverages a grid-based prediction mechanism in the detection head, which is compatible with the global pixel-wise prediction of segmentation tasks.
• Ablation studies compare end-to-end training with step-by-step training and evaluate the impact of different detection frameworks (grid-based vs. region-based).
• The framework is evaluated on the BDD100K dataset using standard metrics: AP for detection, mIoU for segmentation, and IoU for lane detection.

Experimental results

Research questions

• RQ1 Can a single deep learning model achieve state-of-the-art performance on object detection, drivable area segmentation, and lane detection simultaneously in real time?
• RQ2 Does end-to-end multi-task learning outperform separate training of individual tasks in terms of accuracy and efficiency?
• RQ3 Is the grid-based prediction mechanism of single-stage detectors more compatible with semantic segmentation tasks than the region-based mechanism of two-stage detectors?
• RQ4 Can a multi-task network be trained effectively without alternating optimization, preserving performance across all tasks?

Key findings

• YOLOP achieves state-of-the-art performance on all three tasks—object detection (AP: 76.5%), drivable area segmentation (mIoU: 91.5%), and lane detection (IoU: 26.2%)—on the BDD100K dataset.
• The model runs at 41 FPS on a single TITAN XP and 23 FPS on the Jetson TX2, making it the first system to achieve real-time inference on embedded hardware for this multi-task perception setup.
• End-to-end multi-task training performs nearly as well as single-task training, with minimal performance drop (e.g., AP: 76.5% vs. 76.9% for detection alone), while significantly reducing inference time.
• The grid-based detection head in YOLOP enables stable joint training with segmentation heads, whereas the region-based R-CNNP framework suffers from performance degradation when training detection and segmentation tasks together.
• Ablation studies confirm that the grid-based prediction mechanism is more compatible with semantic segmentation tasks than the region-based approach, supporting the design choice of YOLOP.
• The model maintains high accuracy across all tasks without requiring complex alternating optimization, demonstrating the robustness of the end-to-end training paradigm.