[Paper Review] Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation

Jiazhao Zhang, Chenyang Zhu+2 more|arXiv (Cornell University)|Mar 13, 2020

3D Shape Modeling and Analysis36 references4 citations

TL;DR

This paper proposes a fusion-aware 3D point convolution method for online semantic 3D scene segmentation using real-time RGB-D reconstruction. It employs a dual-tree data structure—global coordinate interval tree and local octree—to enable dynamic, efficient 3D convolution that exploits inter-frame temporal coherence, achieving state-of-the-art accuracy at 10 FPS on benchmark datasets like ScanNet.

ABSTRACT

Online semantic 3D segmentation in company with real-time RGB-D reconstruction poses special challenges such as how to perform 3D convolution directly over the progressively fused 3D geometric data, and how to smartly fuse information from frame to frame. We propose a novel fusion-aware 3D point convolution which operates directly on the geometric surface being reconstructed and exploits effectively the inter-frame correlation for high quality 3D feature learning. This is enabled by a dedicated dynamic data structure which organizes the online acquired point cloud with global-local trees. Globally, we compile the online reconstructed 3D points into an incrementally growing coordinate interval tree, enabling fast point insertion and neighborhood query. Locally, we maintain the neighborhood information for each point using an octree whose construction benefits from the fast query of the global tree.Both levels of trees update dynamically and help the 3D convolution effectively exploits the temporal coherence for effective information fusion across RGB-D frames.

Motivation & Objective

• To enable real-time, online semantic 3D scene segmentation during incremental RGB-D reconstruction.
• To address the challenge of performing 3D convolution on time-varying, progressively fused 3D geometry.
• To exploit inter-frame temporal correlation for improved 3D feature learning and segmentation accuracy.
• To design an efficient in-memory data structure that supports dynamic point insertion, neighborhood query, and surface-aware convolution.

Proposed method

• A global coordinate interval tree is used to maintain an incrementally growing spatial organization of online-reconstructed 3D points, enabling fast point insertion and neighborhood queries.
• A local octree is constructed for each point using the global tree’s fast query capability, enabling per-point adaptive convolution with geodesic distance approximation.
• Fusion-aware point convolution integrates intra-frame 2D features and inter-frame 3D features by leveraging point correspondence retrieved from the global tree.
• The method uses weighted convolution kernels based on approximate geodesic distances from the local octree, making the neighborhood selection surface-aware.
• The dynamic dual-tree structure supports real-time updates and enables progressive refinement of semantic labels across frames.
• The system is integrated into an interactive pipeline that performs online segmentation with real-time RGB-D reconstruction.

Experimental results

Research questions

• RQ1 How can 3D convolution be effectively performed on time-varying 3D geometric data during online RGB-D fusion?
• RQ2 What data structure enables efficient dynamic maintenance of online point clouds while supporting fast neighborhood queries for 3D convolution?
• RQ3 How can inter-frame temporal correlation be exploited to improve 3D feature learning and semantic segmentation accuracy?
• RQ4 To what extent does surface-aware neighborhood selection via local octrees improve segmentation performance compared to Euclidean-based methods?
• RQ5 Can a real-time, online semantic 3D segmentation system achieve state-of-the-art accuracy while maintaining interactive frame rates?

Key findings

• The proposed method achieves state-of-the-art semantic segmentation accuracy on the ScanNet benchmark, outperforming both online and offline methods.
• Segmentation accuracy improves significantly over time, with complex objects like chairs and tables showing the most notable gains due to feature fusion across frames.
• The fusion-aware point convolution leads to more separable feature embeddings in the embedding space, as shown by t-SNE visualization of evolving point features.
• Surface-aware convolution using geodesic distance approximation from local octrees consistently outperforms Euclidean-based neighborhood selection across all receptive field sizes.
• The system runs at 10 FPS for online processing and even higher with key-frame-based processing, demonstrating real-time feasibility.
• The dynamic global-local tree structure enables efficient point insertion and neighborhood query, supporting the scalability of the method to large-scale online reconstruction.