[Paper Review] HybridNets: End-to-End Perception Network

Dat Vu, Bao Ngo+1 more|arXiv (Cornell University)|Mar 17, 2022

Advanced Neural Network Applications68 citations

TL;DR

HybridNets presents an end-to-end multi-task perception network for traffic object detection, drivable area segmentation, and lane detection, with optimized head networks, adaptive anchors, and a balanced training strategy, achieving strong accuracy and real-time performance.

ABSTRACT

End-to-end Network has become increasingly important in multi-tasking. One prominent example of this is the growing significance of a driving perception system in autonomous driving. This paper systematically studies an end-to-end perception network for multi-tasking and proposes several key optimizations to improve accuracy. First, the paper proposes efficient segmentation head and box/class prediction networks based on weighted bidirectional feature network. Second, the paper proposes automatically customized anchor for each level in the weighted bidirectional feature network. Third, the paper proposes an efficient training loss function and training strategy to balance and optimize network. Based on these optimizations, we have developed an end-to-end perception network to perform multi-tasking, including traffic object detection, drivable area segmentation and lane detection simultaneously, called HybridNets, which achieves better accuracy than prior art. In particular, HybridNets achieves 77.3 mean Average Precision on Berkeley DeepDrive Dataset, outperforms lane detection with 31.6 mean Intersection Over Union with 12.83 million parameters and 15.6 billion floating-point operations. In addition, it can perform visual perception tasks in real-time and thus is a practical and accurate solution to the multi-tasking problem. Code is available at https://github.com/datvuthanh/HybridNets.

Motivation & Objective

• Study end-to-end multi-task perception networks for driving scenarios.
• Propose efficient segmentation heads and box/class prediction networks based on a weighted bidirectional feature network.
• Introduce automatically customized anchors for each level in the weighted bidirectional feature network.
• Develop an efficient training loss and strategy to balance multi-task optimization.
• Demonstrate multi-task performance on traffic object detection, drivable area segmentation, and lane detection.

Proposed method

• Efficient segmentation head and box/class prediction networks built on a weighted bidirectional feature network.
• Automatically customize anchors at each level of the weighted bidirectional feature network.
• An optimized training loss function and strategy to balance learning across tasks.
• End-to-end framework enabling simultaneous traffic object detection, drivable area segmentation, and lane detection.
• Evaluation on Berkeley DeepDrive Dataset comparing to prior art.

Experimental results

Research questions

• RQ1 Can an end-to-end multi-task perception network improve accuracy across detection, drivable area segmentation, and lane detection compared to prior approaches?
• RQ2 Do weighted bidirectional feature networks with customized anchors improve per-task performance and efficiency?
• RQ3 How effective is the proposed training strategy in balancing multi-task optimization and enabling real-time inference?

Key findings

• HybridNets achieves 77.3 mean Average Precision on the Berkeley DeepDrive Dataset.
• Lane detection achieves 31.6 mean Intersection over Union.
• The model uses 12.83 million parameters and 15.6 billion FLOPs.
• The approach operates in real-time, making it practical for multi-task driving perception.
• Code for HybridNets is available at the provided URL.