[Paper Review] LRF-Net: Learning Local Reference Frames for 3D Local Shape Description and Matching

Angfan Zhu, Jiaqi Yang+2 more|arXiv (Cornell University)|Jan 22, 2020

3D Shape Modeling and Analysis20 references4 citations

TL;DR

LRF-Net proposes a deep learning approach to learn local reference frames (LRFs) for 3D point clouds using a Siamese network with weak supervision, assigning learned weights to neighboring points to improve LRF repeatability and robustness. It outperforms state-of-the-art methods on three public datasets and significantly enhances local shape description and 6-DoF pose estimation performance.

ABSTRACT

The local reference frame (LRF) acts as a critical role in 3D local shape description and matching. However, most of existing LRFs are hand-crafted and suffer from limited repeatability and robustness. This paper presents the first attempt to learn an LRF via a Siamese network that needs weak supervision only. In particular, we argue that each neighboring point in the local surface gives a unique contribution to LRF construction and measure such contributions via learned weights. Extensive analysis and comparative experiments on three public datasets addressing different application scenarios have demonstrated that LRF-Net is more repeatable and robust than several state-of-the-art LRF methods (LRF-Net is only trained on one dataset). In addition, LRF-Net can significantly boost the local shape description and 6-DoF pose estimation performance when matching 3D point clouds.

Motivation & Objective

• To address the limitations of hand-crafted LRFs, which suffer from poor repeatability and robustness under noise, clutter, and occlusion.
• To learn LRFs in a weakly supervised manner, requiring only correspondence relationships between local patches, not ground-truth LRFs or pose annotations.
• To model each neighboring point’s unique contribution to LRF construction through learnable weights, improving stability and invariance.
• To enhance downstream tasks such as local shape description and 6-DoF pose estimation by providing more reliable LRFs.

Proposed method

• LRF-Net uses a Siamese network architecture to learn LRFs from local point cloud patches with weak supervision based on correspondence pairs.
• The z-axis of the LRF is set as the normal vector of the keypoint, ensuring rotational invariance and stability.
• For the x-axis, the network computes rotation-invariant geometric attributes (angle and distance) from neighboring points and uses a multi-layer perceptron (MLP) to predict per-point weights.
• The x-axis is computed as the weighted sum of projection vectors using the learned weights, and the y-axis is derived via cross product with the z-axis.
• The method leverages invariant geometric attributes instead of raw coordinates to ensure rotation invariance and improved generalization.
• The network is trained using a contrastive loss on corresponding local patches, enabling end-to-end learning without ground-truth LRFs.

Experimental results

Research questions

• RQ1 Can a deep learning model learn a more repeatable and robust LRF than hand-crafted methods without requiring ground-truth LRFs or pose annotations?
• RQ2 How do learned point-wise weights compare to heuristic weighting strategies in terms of LRF stability under noise and occlusion?
• RQ3 Does using rotation-invariant geometric attributes instead of raw coordinates improve the robustness and performance of LRF estimation?
• RQ4 To what extent does LRF-Net improve downstream tasks like local shape description and 6-DoF pose estimation compared to SOTA LRF methods?

Key findings

• LRF-Net (Normal + Sum1) achieves the best mean cosine similarity (MeanCos) performance across all three benchmark datasets: BR, UWA3M, and UWAOR.
• On the BR dataset, LRF-Net achieves a MeanCos of 0.999 with the Normal + Sum1 configuration, significantly outperforming other methods.
• The method demonstrates strong robustness to rotation, with stable performance across 30°, 60°, 90°, and 120° rotations, particularly when using the normal vector as guidance.
• LRF-Net maintains stable performance as support radius increases, while other methods degrade, indicating better statistical consistency.
• Visualization shows that learned weights are not necessarily higher for closer points, challenging conventional assumptions in CA- and PSD-based LRFs.
• The integration of learned weights via vector-sum (Sum1) yields superior performance compared to max-pooling or distance-reduction (DR) strategies.