[Paper Review] The Robotic Vision Scene Understanding Challenge

David Hall, Ben Talbot+5 more|arXiv (Cornell University)|Sep 11, 2020

Robotics and Sensor-Based Localization39 references4 citations

TL;DR

This paper introduces a simulation-based robotic vision challenge to standardize evaluation of active scene understanding in indoor environments. It proposes two tasks—Semantic SLAM and Scene Change Detection—using 3D cuboid object maps, tiered difficulty levels, and a novel evaluation metric (OMQ) that combines spatial, semantic, and state quality for robust comparison of robotic vision systems.

ABSTRACT

Being able to explore an environment and understand the location and type of all objects therein is important for indoor robotic platforms that must interact closely with humans. However, it is difficult to evaluate progress in this area due to a lack of standardized testing which is limited due to the need for active robot agency and perfect object ground-truth. To help provide a standard for testing scene understanding systems, we present a new robot vision scene understanding challenge using simulation to enable repeatable experiments with active robot agency. We provide two challenging task types, three difficulty levels, five simulated environments and a new evaluation measure for evaluating 3D cuboid object maps. Our aim is to drive state-of-the-art research in scene understanding through enabling evaluation and comparison of active robotic vision systems.

Motivation & Objective

• To address the lack of standardized, repeatable evaluation for active robotic vision systems in indoor environments.
• To enable comparison of active robotic systems by providing a simulation-based benchmark with standardized tasks and environments.
• To develop a new evaluation metric (OMQ) that quantifies the quality of 3D cuboid object maps in terms of spatial accuracy, semantic label confidence, and change detection state confidence.
• To support ablation studies and broader participation by including tiered difficulty levels and a simple API for robot actions.
• To promote sim-to-real transfer by using high-fidelity simulation platforms that support realistic robotic agency and perception.

Proposed method

• The challenge uses high-fidelity simulation to enable active robot agency, allowing agents to explore environments and build 3D cuboid object maps in real time.
• Two primary tasks are defined: Semantic SLAM (mapping objects with location and class) and Scene Change Detection (detecting added or removed objects).
• The Object Map Quality (OMQ) metric evaluates 3D object maps by combining spatial quality (distance between predicted and ground-truth cuboids), label quality (confidence in correct class), and, for SCD, state quality (confidence in 'added' or 'removed' state).
• False positive cost is computed as the geometric mean of maximum non-background label probability and maximum non-'same' state probability, penalizing overconfident errors.
• The OMQ score is calculated as the harmonic mean of true positive quality scores, normalized by the sum of true positives, false negatives, and weighted false positive costs.
• The system supports a discretized action space via a simple API, enabling reproducible experiments and integration with various robotic perception pipelines.

Experimental results

Research questions

• RQ1 How can active robotic vision systems be evaluated in a standardized, repeatable, and scalable manner that captures the full scope of robotic agency and perception?
• RQ2 What evaluation metric best captures the quality of 3D cuboid object maps in terms of spatial accuracy, semantic correctness, and change detection confidence?
• RQ3 Can a simulation-based benchmark support both advanced roboticists and non-roboticists through tiered difficulty levels and accessible APIs?
• RQ4 How does the inclusion of state quality (e.g., object added/removed) improve the evaluation of dynamic scene understanding tasks like Scene Change Detection?
• RQ5 To what extent can the proposed OMQ metric differentiate between high-quality and overconfident false positive predictions in object mapping?

Key findings

• The proposed OMQ metric successfully integrates spatial, semantic, and state quality into a single evaluation score, enabling fair and nuanced comparison of 3D object maps.
• False positive cost is effectively penalized when both label confidence and state confidence are high, reducing the impact of overconfident errors on overall performance.
• The use of simulation enables repeatable, scalable, and low-cost evaluation of active robotic systems, overcoming limitations of static datasets and real-world hardware constraints.
• The tiered difficulty levels allow for ablation studies and broader participation, including from researchers without extensive robotics infrastructure.
• The challenge supports both Semantic SLAM and Scene Change Detection tasks, with distinct but compatible evaluation protocols tailored to each.
• The evaluation framework is extensible and aligns with existing standards, such as the probabilistic object detection (PrOD) challenges, ensuring compatibility and future adoption.