[Paper Review] ILabel: Interactive Neural Scene Labelling

Shuaifeng Zhi, Edgar Sucar+4 more|arXiv (Cornell University)|Nov 29, 2021

Robotics and Sensor-Based Localization35 references18 citations

TL;DR

iLabel proposes an interactive, real-time 3D scene labelling system that uses a neural field to jointly represent geometry, color, and semantics from ultra-sparse user clicks—requiring no pre-training data. It achieves higher semantic labelling accuracy than pre-trained models with only tens of clicks, surpassing state-of-the-art methods even in low-data regimes like Replica, and supports open-set labelling and hands-free uncertainty-guided annotation.

ABSTRACT

Joint representation of geometry, colour and semantics using a 3D neural field enables accurate dense labelling from ultra-sparse interactions as a user reconstructs a scene in real-time using a handheld RGB-D sensor. Our iLabel system requires no training data, yet can densely label scenes more accurately than standard methods trained on large, expensively labelled image datasets. Furthermore, it works in an 'open set' manner, with semantic classes defined on the fly by the user. ILabel's underlying model is a multilayer perceptron (MLP) trained from scratch in real-time to learn a joint neural scene representation. The scene model is updated and visualised in real-time, allowing the user to focus interactions to achieve efficient labelling. A room or similar scene can be accurately labelled into 10+ semantic categories with only a few tens of clicks. Quantitative labelling accuracy scales powerfully with the number of clicks, and rapidly surpasses standard pre-trained semantic segmentation methods. We also demonstrate a hierarchical labelling variant.

Motivation & Objective

• To enable accurate, dense 3D semantic labelling of scenes with minimal human interaction, especially in low-data or novel environments.
• To eliminate reliance on large, pre-annotated datasets by training a neural field from scratch in real time using only user-provided clicks.
• To support open-set semantic labelling, where users define new categories on-the-fly during scanning.
• To improve user efficiency by enabling real-time visual feedback and interactive correction of predictions.
• To explore hands-free labelling via uncertainty-guided pixel selection, reducing manual click burden.

Proposed method

• Uses a real-time neural field SLAM system to build a joint 3D representation of geometry, appearance, and semantics via a multilayer perceptron (MLP) trained from scratch.
• Integrates sparse user clicks on keyframe images to supervise semantic predictions, which are then propagated across the scene via the neural field's inductive bias.
• Employs real-time rendering of predictions to allow user-in-the-loop interaction, enabling correction of errors and addition of new classes.
• Applies a hierarchical labelling strategy by interpreting semantic outputs as binary tree branches for structured categorization.
• Implements an automatic query generation mode that selects uncertain pixels for user annotation without clicks, using uncertainty sampling.
• Leverages view-based neural representation with spatial continuity to enable generalization across geometric and textural regions.

Experimental results

Research questions

• RQ1 Can a neural field representation enable dense semantic labelling from ultra-sparse user interactions without pre-training on large datasets?
• RQ2 How does labelling accuracy scale with the number of user clicks in a real-time, interactive system compared to pre-trained models?
• RQ3 Can the system support open-set semantic labelling where users define new categories during scanning?
• RQ4 How effective is uncertainty-guided automatic query generation for reducing manual click input while maintaining high accuracy?
• RQ5 Does the neural field's inductive bias enable robust and precise semantic propagation across complex, real-world scenes?

Key findings

• iLabel surpasses the performance of a state-of-the-art pre-trained RGB-D segmentation model (SA-Gate) with only 20 clicks on the Replica dataset, which has a low-data regime with only 7 scenes for fine-tuning.
• On the ScanNet dataset, iLabel reaches similar accuracy to the pre-trained baseline at around 50 clicks and exceeds it by 20% at 120 clicks.
• The automatic query generation mode achieves performance comparable to manual clicking but requires approximately 240 clicks, highlighting the efficiency of direct user interaction.
• Random pixel sampling performs worse than uncertainty-guided selection, especially as more labels are added, demonstrating the importance of intelligent sampling.
• iLabel produces more precise and complete segmentations than SemanticPaint, a comparable interactive system, due to its unified neural field representation.
• The system enables accurate whole-room semantic mesh labelling with only 140 clicks, demonstrating scalability to complex, real-world environments.