[Paper Review] VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition

Yun-Jin Li, Mariia Gladkova+3 more|arXiv (Cornell University)|Mar 21, 2024

Robotics and Sensor-Based Localization4 citations

TL;DR

VXP proposes a self-supervised Voxel-Cross-Pixel framework that establishes 3D-2D feature correspondences between LiDAR point clouds and RGB images, projecting 3D voxel features into 2D image space to align modality-specific features in a shared embedding space. The method achieves state-of-the-art performance on Oxford RobotCar, ViViD++, and KITTI benchmarks, outperforming prior cross-modal retrieval methods with faster inference (7ms for images, 18ms for point clouds).

ABSTRACT

Cross-modal place recognition methods are flexible GPS-alternatives under varying environment conditions and sensor setups. However, this task is non-trivial since extracting consistent and robust global descriptors from different modalities is challenging. To tackle this issue, we propose Voxel-Cross-Pixel (VXP), a novel camera-to-LiDAR place recognition framework that enforces local similarities in a self-supervised manner and effectively brings global context from images and LiDAR scans into a shared feature space. Specifically, VXP is trained in three stages: first, we deploy a visual transformer to compactly represent input images. Secondly, we establish local correspondences between image-based and point cloud-based feature spaces using our novel geometric alignment module. We then aggregate local similarities into an expressive shared latent space. Extensive experiments on the three benchmarks (Oxford RobotCar, ViViD++ and KITTI) demonstrate that our method surpasses the state-of-the-art cross-modal retrieval by a large margin. Our evaluations show that the proposed method is accurate, efficient and light-weight. Our project page is available at: https://yunjinli.github.io/projects-vxp/

Motivation & Objective

• Address the challenge of accurate cross-modal global place recognition between 2D images and 3D LiDAR point clouds, which suffer from domain gaps due to modality differences.
• Overcome limitations of existing methods that rely on preprocessing pipelines (e.g., depth/range image conversion) that degrade uni-modal performance and increase latency.
• Develop a self-supervised method to learn accurate local feature correspondences between voxels and pixels without requiring annotated 3D-2D correspondences.
• Enable robust, real-time place recognition under challenging conditions such as low-light or dynamic lighting, where uni-modal image retrieval fails.
• Design a two-stage training pipeline that jointly optimizes local and global descriptors to improve retrieval accuracy in large-scale settings.

Proposed method

• Introduce a 3D-to-2D projection module that maps 3D voxel features from LiDAR point clouds into 2D image space using geometric projection based on camera intrinsics.
• Establish self-supervised 2D-3D correspondences between projected voxel features and image pixels by leveraging spatial consistency and contrastive learning.
• Train two separate encoders: one for RGB images (e.g., DINO ViT or ResNet) and one for 3D point clouds (e.g., PointNet++-based), both mapping to a shared feature space.
• Apply a two-stage optimization: first minimize local descriptor loss on matched voxel-pixel pairs, then optimize global descriptor loss using contrastive learning on the full image-point cloud pairs.
• Use a pooling layer (e.g., GeM or NetVLAD) on the projected 2D feature map to generate a global image descriptor, while the 3D encoder outputs a global LiDAR descriptor.
• Leverage attention maps from vision transformers (e.g., DINO ViT) to analyze feature importance and validate that salient structures (e.g., buildings) are consistently attended across modalities.

Experimental results

Research questions

• RQ1 Can self-supervised 3D-2D feature correspondence learning effectively bridge the domain gap between images and LiDAR point clouds without annotated supervision?
• RQ2 Does a two-stage training strategy—first optimizing local descriptors and then global descriptors—lead to better cross-modal retrieval performance than end-to-end training?
• RQ3 Can direct processing of raw image and point cloud data outperform methods that require preprocessing into 2.5D representations (e.g., depth or range images)?
• RQ4 How does the proposed method generalize across diverse real-world conditions such as day/night lighting changes, where uni-modal image retrieval fails?
• RQ5 To what extent does the alignment of attention maps between image and projected voxel features indicate effective cross-modal feature learning?

Key findings

• VXP achieves state-of-the-art performance on the Oxford RobotCar benchmark, with a 1.5% absolute improvement in Recall@1 (85.30%) over the previous SOTA, outperforming methods like $LC^{2}$ and Cattaneo et al.
• On the ViViD++ benchmark, VXP achieves a Recall@1 of 84.7%, significantly outperforming prior methods, demonstrating strong generalization across diverse urban and rural environments.
• On the KITTI Odometry dataset, VXP achieves a Recall@1 of 78.6%, showing robustness in large-scale, long-term localization under varying driving conditions.
• VXP achieves fast inference time—7ms for images and 18ms for point clouds on a single RTX 3080, outperforming $LC^{2}$ (17ms and 53ms) due to efficient preprocessing and compact model size (21.7M and 5.9M parameters).
• Visualization of attention maps confirms that salient structures (e.g., buildings) receive high attention in both image and projected voxel features, validating effective cross-modal feature alignment.
• The ablation study shows that the two-stage training pipeline (local then global loss) is critical: removing the local loss leads to a 4.2% drop in Recall@1 on Oxford RobotCar, proving its importance in learning accurate correspondences.