[Paper Review] LION: Linear Group RNN for 3D Object Detection in Point Clouds

Zhe Liu, Jinghua Hou+5 more|arXiv (Cornell University)|Jul 25, 2024

3D Surveying and Cultural Heritage6 citations

TL;DR

LION introduces a window-based 3D backbone using linear group RNNs to enable long-range feature interaction for 3D object detection in sparse point clouds, with a 3D spatial descriptor and voxel generation to boost performance; LION variants achieve state-of-the-art results on multiple autonomous-driving datasets.

ABSTRACT

The benefit of transformers in large-scale 3D point cloud perception tasks, such as 3D object detection, is limited by their quadratic computation cost when modeling long-range relationships. In contrast, linear RNNs have low computational complexity and are suitable for long-range modeling. Toward this goal, we propose a simple and effective window-based framework built on LInear grOup RNN (i.e., perform linear RNN for grouped features) for accurate 3D object detection, called LION. The key property is to allow sufficient feature interaction in a much larger group than transformer-based methods. However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling. To tackle this problem, we simply introduce a 3D spatial feature descriptor and integrate it into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features. To further address the challenge in highly sparse point clouds, we propose a 3D voxel generation strategy to densify foreground features thanks to linear group RNN as a natural property of auto-regressive models. Extensive experiments verify the effectiveness of the proposed components and the generalization of our LION on different linear group RNN operators including Mamba, RWKV, and RetNet. Furthermore, it is worth mentioning that our LION-Mamba achieves state-of-the-art on Waymo, nuScenes, Argoverse V2, and ONCE dataset. Last but not least, our method supports kinds of advanced linear RNN operators (e.g., RetNet, RWKV, Mamba, xLSTM and TTT) on small but popular KITTI dataset for a quick experience with our linear RNN-based framework.

Motivation & Objective

• Motivate and enable long-range feature interaction in 3D point clouds with linear group RNNs to reduce computation costs.
• Enhance linear RNNs for 3D detection by integrating a 3D spatial feature descriptor to capture local geometry.
• Densify sparse foreground features with a voxel generation strategy leveraging auto-regressive properties.
• Demonstrate generalization of LION across multiple linear RNN operators (Mamba, RWKV, RetNet) and datasets.

Proposed method

• Propose LION, a window-based 3D backbone that groups thousands of voxel features for long-range interaction using linear RNNs.
• Introduce a 3D spatial feature descriptor (3D sub-manifold conv + LayerNorm + GELU) to mitigate spatial information loss when flattening to sequences.
• Implement voxel merging and voxel expanding to enable multi-scale feature learning in sparse point clouds.
• Develop a voxel generation strategy that auto-regressively generates foreground voxel features based on high-response areas in the backbone.
• Utilize bi-directional linear RNN operators (Mamba, RWKV, RetNet) in LION blocks with X- and Y-axis window partitions for rich feature interaction.
• Provide a practical framework to replace or augment existing voxel-based backbones with a linear RNN-based approach.

Figure 1: (a) Comparison of different 3D backbones in terms of detection performance on Waymo [ 49 ] , nuScenes [ 4 ] , Argoverse V2 [ 59 ] and ONCE [ 34 ] datasets. Here, we adopt Mamba [ 22 ] as the default operator of our LION. Besides, we present the simplified schematic of DSVT (b) [ 57 ] and o

Experimental results

Research questions

• RQ1 Can linear RNNs with large grouped windows match or surpass transformer-like backbones for 3D object detection in sparse point clouds?
• RQ2 Does adding a 3D spatial feature descriptor improve local spatial modeling for linear group RNNs in 3D backbones?
• RQ3 Can voxel generation via auto-regressive linear RNNs densify sparse foreground regions and boost detection performance?
• RQ4 How well does LION generalize across different linear RNN operators and diverse 3D detection datasets?

Key findings

• LION with different linear RNNs achieves competitive or state-of-the-art performance on Waymo, nuScenes, Argoverse V2, and ONCE datasets.
• A larger group interaction in LION, enabled by linear RNNs, yields improved long-range feature interaction compared to small-group transformer-based methods.
• The 3D spatial feature descriptor effectively mitigates spatial information loss during 1D sequence processing.
• Voxel generation based on auto-regressive properties densifies foreground features and improves discrimination in sparse scenes.
• LION-Mamba achieves state-of-the-art results on multiple datasets, illustrating strong generalization of the LION framework across operators and domains.

Figure 2: The illustration of LION, which mainly consists of several LION blocks, each paired with a voxel generation for feature enhancement and a voxel merging for down-sampling features along the height dimension. $(H,W,D)$ indicates the shape of the 3D feature map, where $H$ , $W$ , and $D$ are