[Paper Review] PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmark

Li Chen, Chonghao Sima+9 more|arXiv (Cornell University)|Mar 21, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

PersFormer proposes an end-to-end monocular 3D lane detection model using a novel Transformer-based spatial feature transformation module that attends to front-view local regions with camera parameters as a reference, enabling accurate BEV feature generation. It achieves state-of-the-art performance on the newly released OpenLane benchmark, significantly outperforming prior methods in 3D lane detection while maintaining strong 2D performance through unified 2D/3D anchor design and multi-task learning.

ABSTRACT

Methods for 3D lane detection have been recently proposed to address the issue of inaccurate lane layouts in many autonomous driving scenarios (uphill/downhill, bump, etc.). Previous work struggled in complex cases due to their simple designs of the spatial transformation between front view and bird's eye view (BEV) and the lack of a realistic dataset. Towards these issues, we present PersFormer: an end-to-end monocular 3D lane detector with a novel Transformer-based spatial feature transformation module. Our model generates BEV features by attending to related front-view local regions with camera parameters as a reference. PersFormer adopts a unified 2D/3D anchor design and an auxiliary task to detect 2D/3D lanes simultaneously, enhancing the feature consistency and sharing the benefits of multi-task learning. Moreover, we release one of the first large-scale real-world 3D lane datasets: OpenLane, with high-quality annotation and scenario diversity. OpenLane contains 200,000 frames, over 880,000 instance-level lanes, 14 lane categories, along with scene tags and the closed-in-path object annotations to encourage the development of lane detection and more industrial-related autonomous driving methods. We show that PersFormer significantly outperforms competitive baselines in the 3D lane detection task on our new OpenLane dataset as well as Apollo 3D Lane Synthetic dataset, and is also on par with state-of-the-art algorithms in the 2D task on OpenLane. The project page is available at https://github.com/OpenPerceptionX/PersFormer_3DLane and OpenLane dataset is provided at https://github.com/OpenPerceptionX/OpenLane.

Motivation & Objective

• To address the limitations of existing 3D lane detection methods that rely on simplistic front-to-BEV feature transformations and lack realistic, large-scale training data.
• To improve BEV feature representation by modeling spatial transformation as a learnable attention-based process that captures contextual interactions across views.
• To unify 2D and 3D lane detection through a shared architecture and auxiliary task, enhancing feature consistency and multi-task learning benefits.
• To release OpenLane, a large-scale, real-world 3D lane dataset with 200,000 frames, 880,000 lane instances, 14 categories, and scene-level annotations to support industrial-grade autonomous driving research.

Proposed method

• Proposes a Perspective Transformer (PersFormer) module that uses deformable self- and cross-attention mechanisms to learn spatial feature transformation from front view to bird’s eye view (BEV), using camera parameters as a reference.
• Introduces a unified 2D/3D anchor design that enables joint detection of 2D and 3D lanes within a single head, promoting feature sharing and consistency.
• Employs an auxiliary binary segmentation head to provide explicit supervision on BEV features, improving feature quality and detection robustness.
• Utilizes a learnable attention mechanism to dynamically attend to relevant local regions in the front-view feature map, aggregating context-aware features for BEV representation.
• Applies differentiable spatial transformation via learnable queries and keys, allowing the model to adaptively focus on salient features during BEV feature generation.
• Releases OpenLane, a large-scale real-world 3D lane dataset with high-quality instance-level annotations, scene tags, and closed-in-path object annotations to support realistic benchmarking.

Experimental results

Research questions

• RQ1 Can a learnable, attention-based spatial transformation module outperform traditional inverse perspective mapping (IPM) in generating BEV features for 3D lane detection?
• RQ2 Does joint 2D/3D lane detection with a unified anchor design improve feature consistency and detection accuracy compared to separate 2D and 3D heads?
• RQ3 To what extent does explicit BEV supervision via an auxiliary segmentation head enhance 3D lane detection performance?
• RQ4 How does the proposed method generalize across diverse real-world driving scenarios, including uphill/downhill and complex road structures?
• RQ5 Can a large-scale, real-world 3D lane dataset like OpenLane effectively support the development and benchmarking of industrial-grade 3D lane detection models?

Key findings

• PersFormer achieves a 92.9% F-Score on the Apollo 3D Lane Synthetic dataset’s balanced scenes, outperforming prior methods including 3D-LaneNet and Gen-LaneNet.
• On the rare scene subset of Apollo 3D Lane Synthetic, PersFormer achieves 87.5% F-Score, surpassing the previous best method (CLGo at 86.1%) by 1.4 percentage points.
• On the OpenLane benchmark, PersFormer achieves a 3D F-Score of 47.79% on a 300-segment subset, representing a 6.02% improvement over the baseline when using the complete ablation setup.
• The ablation study confirms that the combination of unified anchors, multi-task learning, and the Perspective Transformer yields a 4.9% improvement over baseline 3D detection, with an additional 1.13% gain from the auxiliary segmentation head.
• PersFormer improves 2D lane detection F-Score by 9.7% on OpenLane compared to the baseline, demonstrating that enhanced BEV features benefit 2D tasks as well.
• Qualitative results show PersFormer detects subtle curbside lanes and maintains parallel topology in uphill/downhill scenarios, while competing methods fail due to planar assumptions.