[Paper Review] CVTNet: A Cross-View Transformer Network for Place Recognition Using LiDAR Data

Junyi Ma, Guangming Xiong+2 more|arXiv (Cornell University)|Feb 3, 2023

Robotics and Sensor-Based Localization4 citations

TL;DR

CVTNet proposes a cross-view transformer network that fuses multi-layer range image views (RIVs) and bird’s eye views (BEVs) from LiDAR data to generate yaw-angle-invariant global descriptors for robust, real-time place recognition. By leveraging intra- and inter-transformer modules to model correlations within and between views, CVTNet achieves state-of-the-art performance with inference speeds exceeding 29 Hz—faster than typical LiDAR frame rates—on large-scale outdoor datasets including NCLT, KITTI, and a self-recorded dataset.

ABSTRACT

LiDAR-based place recognition (LPR) is one of the most crucial components of autonomous vehicles to identify previously visited places in GPS-denied environments. Most existing LPR methods use mundane representations of the input point cloud without considering different views, which may not fully exploit the information from LiDAR sensors. In this paper, we propose a cross-view transformer-based network, dubbed CVTNet, to fuse the range image views (RIVs) and bird's eye views (BEVs) generated from the LiDAR data. It extracts correlations within the views themselves using intra-transformers and between the two different views using inter-transformers. Based on that, our proposed CVTNet generates a yaw-angle-invariant global descriptor for each laser scan end-to-end online and retrieves previously seen places by descriptor matching between the current query scan and the pre-built database. We evaluate our approach on three datasets collected with different sensor setups and environmental conditions. The experimental results show that our method outperforms the state-of-the-art LPR methods with strong robustness to viewpoint changes and long-time spans. Furthermore, our approach has a good real-time performance that can run faster than the typical LiDAR frame rate. The implementation of our method is released as open source at: https://github.com/BIT-MJY/CVTNet.

Motivation & Objective

• To address the limitation of existing LiDAR-based place recognition (LPR) methods that use single-view or non-optimized representations of point clouds.
• To exploit the complementary information from multi-layer RIVs and BEVs by modeling their latent alignment through a novel transformer architecture.
• To develop a yaw-angle-invariant descriptor that ensures robust place recognition under significant viewpoint changes, including opposite driving directions.
• To achieve real-time inference performance suitable for online autonomous driving applications.

Proposed method

• The method projects 3D LiDAR point clouds into multi-layer RIVs and BEVs to capture spatial and range information across different depths and heights.
• It uses a novel cross-view transformer network (CVTNet) that applies intra-attention within each view (RIV and BEV) to extract intra-view correlations.
• Inter-attention is applied between the aligned RIV and BEV features to model cross-view dependencies and enhance feature discrimination.
• The network generates a single, global, yaw-angle-invariant descriptor end-to-end for each LiDAR scan using a learnable class token.
• The architecture is designed to be strictly invariant to yaw rotation by leveraging the geometric correspondence between RIV and BEV projections.
• Place recognition is performed via descriptor matching between the current query scan and a pre-built database using FAISS for efficient retrieval.

Experimental results

Research questions

• RQ1 Can fusing multi-layer RIVs and BEVs through a transformer-based architecture improve feature representation for LiDAR-based place recognition?
• RQ2 How effective is the proposed cross-view transformer in achieving yaw-angle invariance under large viewpoint changes?
• RQ3 Does the use of multi-layer inputs enhance feature discrimination compared to single-channel representations?
• RQ4 Can the proposed method achieve real-time inference performance suitable for online autonomous driving systems?
• RQ5 How does the proposed method compare to state-of-the-art LPR methods in terms of accuracy and robustness across diverse datasets?

Key findings

• CVTNet achieves state-of-the-art performance on the NCLT, KITTI, and self-recorded datasets, with an average recall@1 of 0.907 on the self-recorded dataset using multi-layer inputs.
• The method demonstrates strong yaw-angle invariance, maintaining high performance (Recall@1 > 0.85) across all 30° yaw rotation steps, outperforming MinkLoc3D and PointNetVLAD, which degrade significantly with increasing rotation.
• CVTNet runs at 29.85 Hz (33.50 ms per inference), exceeding the typical 10 Hz LiDAR frame rate, making it suitable for real-time deployment.
• The ablation study confirms that multi-layer input improves performance over single-channel input, with a 0.041 gain in Recall@1 on the self-recorded dataset.
• The inter-transformer module contributes more to performance gain than the intra-transformer, indicating that cross-view correlation is critical for robust descriptor learning.
• The model has only 21.38 million parameters, indicating low memory consumption and high efficiency for embedded deployment.