[Paper Review] ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects

Dhruv Batra, Aaron Gokaslan+6 more|arXiv (Cornell University)|Jun 23, 2020

Multimodal Machine Learning Applications18 references129 citations

TL;DR

The paper provides consensus recommendations for ObjectNav evaluation, embodiment, and environments, and details their instantiation in Habitat and RoboTHOR 2020 challenges.

ABSTRACT

We revisit the problem of Object-Goal Navigation (ObjectNav). In its simplest form, ObjectNav is defined as the task of navigating to an object, specified by its label, in an unexplored environment. In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e.g., find a chair, by navigating to it. As the community begins to show increased interest in semantic goal specification for navigation tasks, a number of different often-inconsistent interpretations of this task are emerging. This document summarizes the consensus recommendations of this working group on ObjectNav. In particular, we make recommendations on subtle but important details of evaluation criteria (for measuring success when navigating towards a target object), the agent's embodiment parameters, and the characteristics of the environments within which the task is carried out. Finally, we provide a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020 http://embodied-ai.org .

Motivation & Objective

• Define precise evaluation criteria for ObjectNav (navigation to an object) to enable fair comparisons across approaches.
• Clarify agent embodiment (actions, observations) and environmental characteristics used in ObjectNav.
• Recommend suitable datasets, 3D environments, and platforms to benchmark ObjectNav in realistic settings.
• Promote consistency and systematic benchmarking for ObjectNav research by detailing implementations used in challenges.

Proposed method

• Propose a structured ObjectNav task definition including Object Finding and Evaluation, Embodiment, and Environments.
• Define success criteria for ObjectNav (Intentionality, Validity, Proximity, Visibility) and an ObjectNav-SPL metric with discussion of its known issues.
• Recommend an embodiment design with discretized actions and realistic sensors, avoiding full robotic dynamics but preserving planning difficulty.
• Advocate use of large, photo-realistic 3D scans (Matterport3D, Gibson, AI2-THOR/RoboTHOR) and object-class-based goals (e.g., chair, table).
• Describe how to instantiate these recommendations in Habitat 2020 and RoboTHOR 2020 challenges, including agent specs, sensor sets, and phase structure.

Experimental results

Research questions

• RQ1 What constitutes a robust and fair evaluation protocol for ObjectNav across diverse embodiments and environments?
• RQ2 How should success be defined for ObjectNav to reflect true navigational ability to reach an object?
• RQ3 What embodiment and observation configurations best balance realism with tractable research benchmarks in ObjectNav?
• RQ4 Which environments and object classes provide realistic, transferable benchmarks for ObjectNav research?
• RQ5 How can challenges (Habitat and RoboTHOR) operationalize these recommendations for consistent benchmarking?

Key findings

• Propose a multi-criteria success definition: Intentionality, Validity, Proximity, and Visibility within a defined success zone.
• Highlight SPL limitations: non-differentiated failures, high variance, no penalty for turning, and cross-dataset comparison issues; call for new metrics in the future.
• Recommend discretized action spaces and realistic sensors (RGB-D, GPS+Compass) without full robotic dynamics.
• Advocate using photo-realistic 3D scans (Matterport3D, Gibson, AI2-THOR/RoboTHOR) with object-class goals and diverse indoor environments.
• Provide concrete instantiations for Habitat 2020 (scene/object selection, agent specs, and challenge phases) and RoboTHOR 2020 (simulation-to-real setup, phases, and evaluation criteria).
• Document known collision dynamics issues (sliding) and remedies (disable sliding) to prevent “cheating” in simulation.