[Paper Review] FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB Image

Jingwei Huang, Yichao Zhou+2 more|arXiv (Cornell University)|Mar 29, 2019

Advanced Vision and Imaging40 references4 citations

TL;DR

FrameNet proposes a deep learning framework that predicts local canonical frames (LCFs) on 3D surfaces from a single RGB image by jointly regressing surface normals and tangent principal directions. By introducing a joint loss that enforces geometric consistency, the method improves normal estimation accuracy and enables robust 3D-aware image manipulation, achieving state-of-the-art performance on NYUv2 and ScanNet with consistent, detailed surface normal predictions and improved feature matching for AR applications.

ABSTRACT

In this work, we introduce the novel problem of identifying dense canonical 3D coordinate frames from a single RGB image. We observe that each pixel in an image corresponds to a surface in the underlying 3D geometry, where a canonical frame can be identified as represented by three orthogonal axes, one along its normal direction and two in its tangent plane. We propose an algorithm to predict these axes from RGB. Our first insight is that canonical frames computed automatically with recently introduced direction field synthesis methods can provide training data for the task. Our second insight is that networks designed for surface normal prediction provide better results when trained jointly to predict canonical frames, and even better when trained to also predict 2D projections of canonical frames. We conjecture this is because projections of canonical tangent directions often align with local gradients in images, and because those directions are tightly linked to 3D canonical frames through projective geometry and orthogonality constraints. In our experiments, we find that our method predicts 3D canonical frames that can be used in applications ranging from surface normal estimation, feature matching, and augmented reality.

Motivation & Objective

• To learn local canonical frames (LCFs) on 3D surfaces from a single RGB image without explicit 3D supervision.
• To improve surface normal estimation by incorporating geometric priors through projected tangent principal directions.
• To enable consistent 3D-aware image manipulation, such as rigid or deformable object insertion in augmented reality.
• To reduce domain gap issues between synthetic and real-world datasets by leveraging geometric supervision during training.

Proposed method

• Trains a deep neural network using a joint loss function that regresses both surface normals and projected tangent principal directions from a single RGB image.
• Uses a multi-task learning setup where the network outputs 13-dimensional features at the final layer, including normal and tangent direction predictions.
• Applies a fixed learning rate of 1e-5 and processes input images at 320x240 resolution during training.
• Leverages geometric consistency between predicted normals and tangent directions to improve supervision and reduce prediction errors.
• Employs perspective rectification in feature matching using SIFT, enhancing correspondence accuracy compared to standard SIFT.
• Validates performance on real-world datasets (NYUv2, ScanNet) after pre-training on synthetic data (SunCG), demonstrating zero-shot generalization.

Experimental results

Research questions

• RQ1 Can local canonical frames be effectively predicted from a single RGB image using deep learning without explicit 3D supervision?
• RQ2 How does joint supervision of surface normals and tangent principal directions improve normal estimation accuracy?
• RQ3 To what extent does the proposed method generalize across domain shifts between synthetic (SunCG) and real-world (NYUv2, ScanNet) datasets?
• RQ4 Can the predicted canonical frames enable more accurate and consistent 3D-aware image manipulation in augmented reality?
• RQ5 Does the inclusion of geometric priors via tangent direction supervision lead to better feature matching and reduced prediction errors?

Key findings

• The DORN-Ours model trained on ScanNet achieved the best performance on NYUv2, with a mean error of 21.99°, median of 14.29°, and RMSE of 29.87°, outperforming all baselines.
• On SunCG, all models with the joint loss (e.g., DORN-Ours) reduced mean error from 12.90° to 12.38° and RMSE from 24.12° to 23.34°, showing consistent improvement.
• Visualizations confirmed that predicted surface normals had fewer errors at object boundaries and captured finer details, especially on small objects.
• Projected tangent principal directions were visually consistent with ground truth and aligned with human intuition, indicating effective geometric supervision.
• Feature matching using SIFT with perspective rectification produced more correct correspondences than standard SIFT, as shown in quantitative and qualitative comparisons.
• Augmented reality results demonstrated locally consistent perspective alignment when inserting 3D objects or images, confirming the utility of the predicted canonical frames.