[Paper Review] NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

Jiaming Sun, Yiming Xie+3 more|arXiv (Cornell University)|Apr 1, 2021

Advanced Vision and Imaging54 references4 citations

TL;DR

NeuralRecon proposes a learning-based, real-time 3D reconstruction system that directly predicts sparse TSDF volumes for local video fragments using 3D sparse convolutions and a GRU-based fusion module, enabling dense, coherent, and accurate 3D geometry reconstruction from monocular video at 33 FPS—outperforming prior methods in speed and accuracy while being the first to achieve real-time, coherent reconstruction using deep learning.

ABSTRACT

We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time.

Motivation & Objective

• Address the limitations of depth-based reconstruction pipelines that estimate depth maps per frame and fuse them later, which suffer from inconsistency and redundancy.
• Enable real-time, dense, and globally coherent 3D reconstruction from monocular video streams using deep learning.
• Overcome scale inconsistencies and geometric artifacts caused by independent single-view depth estimation by jointly reconstructing surfaces within local video fragments.
• Introduce a differentiable, recurrent fusion mechanism that conditions current reconstruction on previously reconstructed global geometry to ensure temporal and spatial consistency.
• Achieve real-time performance (33 FPS) while maintaining high reconstruction accuracy, surpassing existing learning-based and non-learning-based methods.

Proposed method

• Process monocular video frames with a 2D image encoder to extract features, which are then unprojected into a 3D sparse feature volume.
• Apply a coarse-to-fine 3D sparse convolutional network to predict sparse TSDF volumes for each local video fragment, learning local smoothness and global shape priors.
• Use a gated recurrent unit (GRU) to fuse features from the current fragment with the hidden state of previously reconstructed geometry, enabling context-aware, sequential reconstruction.
• Fusion is performed either within the occupied volume (OCC) or the fragment’s bounding volume (FBV), with FBV improving context integration and reducing artifacts.
• Leverage the GRU-based fusion module to selectively integrate consistent geometric information across time, improving robustness and completeness over linear or average fusion.
• Reconstruct the final surface using Marching Cubes on the fused TSDF volume, enabling direct use in downstream tasks like 3D detection and neural rendering.

Figure 1: Comparison between depth-based 3D reconstruction methods and the proposed method. In depth-based methods, key-frame depths are estimated separately from each key frame, and later fused into a TSDF volume. In the proposed method, the TSDF volume is directly predicted with all the key frames

Experimental results

Research questions

• RQ1 Can a deep learning-based system jointly reconstruct and fuse 3D geometry in real-time from monocular video, avoiding the pitfalls of separate depth estimation and fusion?
• RQ2 How effective is a GRU-based recurrent fusion module in maintaining global consistency and improving reconstruction quality compared to conventional linear or average fusion?
• RQ3 Does reconstructing within local video fragments (rather than full scenes) improve geometric coherence and reduce noise from distant or irrelevant views?
• RQ4 Can a sparse, coarse-to-fine 3D CNN architecture achieve high-fidelity reconstruction while maintaining real-time inference speed on consumer hardware?
• RQ5 To what extent does using feature fusion instead of direct TSDF fusion improve reconstruction accuracy and completeness?

Key findings

• NeuralRecon achieves 33 key frames per second, approximately 10× faster than Atlas, the previous state-of-the-art volume-based method, while maintaining superior accuracy.
• On the ScanNet dataset, the method achieves a 3D F-score of 0.516 with 9-view fragments, outperforming both depth-based and volume-based baselines in reconstruction quality.
• Using GRU-based fusion instead of average or linear fusion improves recall by 4% (from 0.535 to 0.572) and enhances mesh completeness, especially on challenging surfaces like walls and furniture.
• Fusing features within the fragment’s bounding volume (FBV) yields 6.5% higher F-score than within the occupied volume (OCC), demonstrating the benefit of contextual integration at boundaries.
• The method produces sharper, more coherent geometry than Atlas and depth-based baselines, with fewer artifacts and consistent scale across views, as shown in qualitative comparisons.
• The ablation study confirms that increasing fragment length from 5 to 9 views improves F-score by over 2%, indicating that more views per fragment enhance reconstruction accuracy.

Figure 2: NeuralRecon architecture. NeuralRecon predicts TSDF with a three-level coarse-to-fine approach that gradually increases the density of sparse voxels. Key-frame images in the local fragment are first passed through the image backbone to extract the multi-level features. These image features