[Paper Review] DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching

Shuzhe Wang, Juho Kannala+1 more|arXiv (Cornell University)|Jun 21, 2023

Robotics and Sensor-Based Localization4 citations

TL;DR

DGC-GNN proposes a descriptor-free 2D-3D matching method that leverages geometric and color cues through a global-to-local Graph Neural Network (GNN), using a cluster-based transformer to guide local matching. It achieves state-of-the-art performance, doubling the accuracy of the prior best descriptor-free method (GoMatch) and significantly narrowing the gap with descriptor-based approaches.

ABSTRACT

Matching 2D keypoints in an image to a sparse 3D point cloud of the scene without requiring visual descriptors has garnered increased interest due to its low memory requirements, inherent privacy preservation, and reduced need for expensive 3D model maintenance compared to visual descriptor-based methods. However, existing algorithms often compromise on performance, resulting in a significant deterioration compared to their descriptor-based counterparts. In this paper, we introduce DGC-GNN, a novel algorithm that employs a global-to-local Graph Neural Network (GNN) that progressively exploits geometric and color cues to represent keypoints, thereby improving matching accuracy. Our procedure encodes both Euclidean and angular relations at a coarse level, forming the geometric embedding to guide the point matching. We evaluate DGC-GNN on both indoor and outdoor datasets, demonstrating that it not only doubles the accuracy of the state-of-the-art visual descriptor-free algorithm but also substantially narrows the performance gap between descriptor-based and descriptor-free methods.

Motivation & Objective

• To address the limitations of descriptor-free 2D-3D matching, including poor robustness and performance degradation compared to descriptor-based methods.
• To explore whether geometric cues alone are sufficient for reliable matching, especially in geometrically ambiguous scenes.
• To improve matching robustness by integrating both geometric and color information in a hierarchical, global-to-local GNN framework.
• To reduce memory footprint and privacy risks associated with storing high-dimensional visual descriptors in 3D scene models.
• To develop a method that generalizes across diverse indoor and outdoor scenes and keypoint detectors without performance degradation.

Proposed method

• The method employs a global-to-local GNN that first encodes global geometric and color features using a cluster-based transformer to form a geometric embedding.
• It computes a distance-angular embedding to capture both Euclidean and angular relations between keypoints at a coarse level.
• Local matching is guided by the global embedding, with a clustering mechanism that constrains information flow within local clusters.
• The network fuses RGB color features with geometric coordinates and uses a learnable attention mechanism to refine local point representations.
• The model is trained end-to-end to predict 2D-3D correspondences without requiring visual descriptors, using differentiable geometric optimization.
• It uses K-Means and Mean-Shift clustering for global cluster assignment, with ablation studies confirming robustness to clustering algorithm choice.

Figure 1 : 2D-3D matching (shown by green lines) with the proposed DGC-GNN and GoMatch [ 54 ] . In this example, DGC-GNN obtains 78 correct matches with 0.02 meters camera translation and 0.24 ∘ rotation errors, while GoMatch finds only 17 inliers with a pose error of 0.37 meters and 4.37 ∘ .

Experimental results

Research questions

• RQ1 Can geometric cues alone reliably distinguish between similar structures in 2D-3D matching, or is additional information like color necessary?
• RQ2 How can global geometric structure be effectively encoded and used to guide local matching in a descriptor-free setting?
• RQ3 Does incorporating color information improve matching accuracy without compromising privacy or increasing memory overhead?
• RQ4 How does the performance of a descriptor-free method compare to descriptor-based methods when both use the same 3D point cloud?
• RQ5 Can the proposed method generalize across different datasets, keypoint detectors, and scene types?

Key findings

• DGC-GNN doubles the accuracy of the state-of-the-art descriptor-free method, GoMatch, on both indoor and outdoor datasets.
• On the 7Scenes dataset, DGC-GNN achieves a median translation error of 1.43 cm and rotation error of 13.43° when trained on ScanNet and evaluated with SIFT, outperforming GoMatch.
• The model reduces the performance gap between descriptor-free and descriptor-based methods, achieving near-state-of-the-art accuracy in pose estimation.
• Ablation studies confirm that geometric embedding (G. Emb.) is superior to label-based embedding (G. Label), highlighting the importance of geometric cues in the global representation.
• The method is robust to different clustering algorithms (K-Means vs. Mean-Shift), with only minor performance differences observed.
• The model generalizes well across datasets and keypoint detectors, with only minor performance drops when using SuperPoint instead of SIFT.

Figure 2 : Pipeline overview. For keypoints from the 2D image and 3D points from the point cloud, the proposed DGC-GNN (1) considers the bearing vectors and the color at each bearing vector as input. (2) It extracts the point-wise position and color features with two separate encoders and mixes the