[Paper Review] SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration

Antoine Guédon, Pascal Monasse+1 more|arXiv (Cornell University)|Aug 22, 2022

Advanced Vision and Imaging4 citations

TL;DR

SCONE proposes a deep learning method for Next Best View (NBV) prediction in large, unknown 3D environments by combining volumetric occupancy prediction with Monte Carlo integration of visibility scores. It uses a neural network to predict occupancy probabilities and visibility via self-attention, enabling accurate surface coverage optimization with free 6D camera motion, outperforming prior methods on both object-level and large-scale scene reconstruction tasks.

ABSTRACT

Next Best View computation (NBV) is a long-standing problem in robotics, and consists in identifying the next most informative sensor position(s) for reconstructing a 3D object or scene efficiently and accurately. Like most current methods, we consider NBV prediction from a depth sensor like Lidar systems. Learning-based methods relying on a volumetric representation of the scene are suitable for path planning, but have lower accuracy than methods using a surface-based representation. However, the latter do not scale well with the size of the scene and constrain the camera to a small number of poses. To obtain the advantages of both representations, we show that we can maximize surface metrics by Monte Carlo integration over a volumetric representation. In particular, we propose an approach, SCONE, that relies on two neural modules: The first module predicts occupancy probability in the entire volume of the scene. Given any new camera pose, the second module samples points in the scene based on their occupancy probability and leverages a self-attention mechanism to predict the visibility of the samples. Finally, we integrate the visibility to evaluate the gain in surface coverage for the new camera pose. NBV is selected as the pose that maximizes the gain in total surface coverage. Our method scales to large scenes and handles free camera motion: It takes as input an arbitrarily large point cloud gathered by a depth sensor as well as camera poses to predict NBV. We demonstrate our approach on a novel dataset made of large and complex 3D scenes.

Motivation & Objective

• To address the limitations of existing NBV methods that either lack scalability (surface-based) or accuracy (volumetric) in large and complex 3D scenes.
• To enable effective path planning and 3D reconstruction in unknown environments with free 6D camera motion, not constrained to a sphere around the object.
• To combine the scalability of volumetric representations with the accuracy of surface-based metrics by using Monte Carlo integration over volumetric occupancy fields.
• To develop a learnable, differentiable framework that predicts surface coverage gain for any camera pose using deep implicit functions and self-attention.
• To create a new large-scale, complex 3D scene dataset under CC license for benchmarking NBV methods with free camera motion.

Proposed method

• A deep implicit network predicts occupancy probability in the 3D scene volume from input point clouds, enabling high-resolution, scalable geometry estimation.
• A second neural module uses self-attention to predict visibility likelihood of sampled 3D points across all directions based on occupancy probability and camera history.
• Monte Carlo integration is applied over the volumetric occupancy field to estimate surface coverage gain for any candidate camera pose.
• The NBV is selected as the pose that maximizes the estimated gain in total visible surface area.
• The method uses a grid of local point clouds instead of a global embedding to scale efficiently to large scenes.
• A continuous IoU metric and Kullback-Leibler divergence loss are used to train the occupancy and visibility prediction modules.

Experimental results

Research questions

• RQ1 Can a volumetric deep learning approach achieve high accuracy in surface coverage estimation while scaling to large 3D scenes?
• RQ2 Can Monte Carlo integration over a volumetric occupancy field effectively optimize surface coverage gain for NBV selection?
• RQ3 Does incorporating camera history improve NBV selection performance in free 6D motion scenarios?
• RQ4 Can a model trained on single-object datasets generalize to complex, large-scale 3D scenes with free camera motion?
• RQ5 How does the proposed method compare to state-of-the-art NBV methods in terms of surface coverage and reconstruction quality?

Key findings

• SCONE outperforms state-of-the-art methods on the ShapeNet dataset, achieving a mean surface coverage of 0.625 on the test set, surpassing prior learning-based approaches.
• On large-scale scenes such as the Eiffel Tower and Fushimi Castle, SCONE achieves mean surface coverage of 0.741 and 0.802, respectively, demonstrating strong generalization.
• The ablation study shows that neighborhood features are critical for occupancy prediction, with MSE dropping from 0.0816 to 0.0397 when included.
• The full model with both occupancy and visibility modules reduces Kullback-Leibler divergence loss by 30% compared to baseline, indicating better distributional estimation of coverage gains.
• Camera history features marginally improve NBV selection accuracy, particularly in identifying the maximum gain, but do not significantly affect the overall coverage distribution.
• The method generalizes well to complex scenes beyond its training domain, as validated on a novel dataset of large-scale 3D scenes under CC license.