[Paper Review] 3D-MAN: 3D Multi-frame Attention Network for Object Detection

Zetong Yang, Yin Zhou+2 more|arXiv (Cornell University)|Mar 30, 2021

Advanced Neural Network Applications35 references4 citations

TL;DR

3D-MAN proposes a 3D multi-frame attention network that improves 3D object detection by fusing features from multiple LiDAR frames using a memory bank and a multi-view alignment and aggregation module. It achieves state-of-the-art performance on the Waymo Open Dataset, reaching 74.3% AP on vehicles with 16 input frames, outperforming both single-frame and multi-frame baselines by significant margins.

ABSTRACT

3D object detection is an important module in autonomous driving and robotics. However, many existing methods focus on using single frames to perform 3D detection, and do not fully utilize information from multiple frames. In this paper, we present 3D-MAN: a 3D multi-frame attention network that effectively aggregates features from multiple perspectives and achieves state-of-the-art performance on Waymo Open Dataset. 3D-MAN first uses a novel fast single-frame detector to produce box proposals. The box proposals and their corresponding feature maps are then stored in a memory bank. We design a multi-view alignment and aggregation module, using attention networks, to extract and aggregate the temporal features stored in the memory bank. This effectively combines the features coming from different perspectives of the scene. We demonstrate the effectiveness of our approach on the large-scale complex Waymo Open Dataset, achieving state-of-the-art results compared to published single-frame and multi-frame methods.

Motivation & Objective

• To address the ambiguity in single-frame 3D object detection caused by partial LiDAR point clouds.
• To improve detection accuracy for partially visible or fast-moving objects by leveraging temporal information from multiple frames.
• To design an efficient, real-time capable multi-frame 3D detection framework that outperforms existing point concatenation and recurrent methods.
• To demonstrate the effectiveness of attention-based feature aggregation across multiple frames in complex, large-scale autonomous driving scenarios.

Proposed method

• A fast single-frame detector (FSD) is used to generate high-quality box proposals and feature maps in real time, employing a Hungarian-matching-based loss and max-pooling NMS.
• Proposals and corresponding feature maps from previous frames are stored in a memory bank to preserve multi-perspective representations over time.
• A multi-view alignment and aggregation module (MVAA) uses attention mechanisms to extract relevant features from the memory bank for each target proposal.
• The alignment stage uses target proposals as queries to retrieve features from stored frames, while the aggregation stage fuses features across frames per proposal.
• An auxiliary cross-view loss is introduced to improve alignment quality by encouraging attention weights to match ground-truth correspondences.
• Ego-motion correction is applied to align point clouds from earlier frames to the coordinate system of the current frame, enabling consistent feature fusion.

Figure 1: Upper row: Potential detections given LiDAR from a single frame demonstrating ambiguity between many reasonable predictions. Lower row: After merging the points aligned across 4 frames, there is more certainty for the correct box prediction.

Experimental results

Research questions

• RQ1 Can attention-based feature aggregation across multiple LiDAR frames improve 3D object detection accuracy compared to single-frame or point concatenation methods?
• RQ2 How does the proposed 3D-MAN model perform on long time horizons and for fast-moving objects, where point misalignment is a major challenge?
• RQ3 Does the use of a Hungarian-matching-based loss and max-pooling NMS in the fast single-frame detector lead to higher-quality proposals that enhance downstream multi-frame fusion?
• RQ4 To what extent does the auxiliary cross-view loss improve the alignment of features from different frames in the memory bank?
• RQ5 Can the model maintain or improve performance when increasing the number of input frames, especially for challenging velocity categories?

Key findings

• 3D-MAN achieves 74.3% AP on the Waymo Open Dataset for vehicle detection with 16 input frames, a significant improvement over the 68.2% AP of the single-frame baseline.
• The model outperforms the point concatenation method by 2.0% AP and the Relation self-attention method by 2.4% AP on the 4-frame setting.
• Performance steadily increases with more input frames, rising from 72.5% AP (4 frames) to 74.3% AP (16 frames), indicating strong scalability over time.
• The model improves detection for fast-moving vehicles, increasing AP from 71.90% (1-frame) to 79.2% (16-frames), unlike baseline methods that degrade with longer horizons.
• The cross-view loss improves AP by 1.8% over no auxiliary loss and by 1.1% over the correspondence loss, demonstrating its effectiveness in enhancing feature alignment.
• The Hungarian-matching-based loss in the FSD yields the highest AP (72.5%) among all compared supervision strategies, validating its superiority in proposal quality.

Figure 2: Framework for 3D-MAN: 3D multi-frame attention network. Given the point cloud for a target frame $t$ , a fast single-frame detector first generates box proposals. These proposals (box parameters) with the feature map (last layer of the backbone network) are inserted into a memory bank that