[Paper Review] Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances

Jacob Krantz, Stefan Lee+3 more|arXiv (Cornell University)|Nov 29, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Instance-Specific Image Goal Navigation (InstanceImageNav), a new benchmark for embodied AI that requires agents to navigate to specific object instances using goal images taken from arbitrary viewpoints and camera parameters, independent of the agent’s sensors. The method uses focused, high-coverage images of object instances in HM3D environments, and a model-free RL baseline achieves only 5.5% success rate, highlighting the task's challenge and need for future research.

ABSTRACT

We consider the problem of embodied visual navigation given an image-goal (ImageNav) where an agent is initialized in an unfamiliar environment and tasked with navigating to a location 'described' by an image. Unlike related navigation tasks, ImageNav does not have a standardized task definition which makes comparison across methods difficult. Further, existing formulations have two problematic properties; (1) image-goals are sampled from random locations which can lead to ambiguity (e.g., looking at walls), and (2) image-goals match the camera specification and embodiment of the agent; this rigidity is limiting when considering user-driven downstream applications. We present the Instance-specific ImageNav task (InstanceImageNav) to address these limitations. Specifically, the goal image is 'focused' on some particular object instance in the scene and is taken with camera parameters independent of the agent. We instantiate InstanceImageNav in the Habitat Simulator using scenes from the Habitat-Matterport3D dataset (HM3D) and release a standardized benchmark to measure community progress.

Motivation & Objective

• To address the lack of standardization and realism in existing ImageNav benchmarks, which often use randomly sampled, ambiguous image goals.
• To overcome the limitation of tying image goals to the agent’s sensor and embodiment, which restricts real-world applicability.
• To improve task clarity and application relevance by focusing goal images on specific object instances in natural, context-rich views.
• To establish a standardized benchmark with a public leaderboard to enable fair evaluation and community progress in instance-specific visual navigation.
• To release a large-scale episode dataset with 7M training and 2.3K validation episodes for training and evaluating navigation policies.

Proposed method

• Goal images are generated by sampling camera viewpoints radially around object instances, with thresholds on object coverage and frame coverage to ensure clear, natural views.
• The task is instantiated in the Habitat-Sim environment using HM3D scenes annotated with HM3DSEM semantic segmentation.
• Starting poses are sampled via rejection sampling to ensure finite geodesic paths and non-trivial navigation, with a minimum ratio of geodesic to Euclidean distance of 1.05.
• A model-free reinforcement learning baseline is trained using PPO with a 2-stream ResNet-18 encoder for RGB, depth, and goal image features, plus LSTM-based memory and action head.
• Features from RGB, depth, goal image, GPS, and heading are concatenated and processed through a two-layer LSTM before action prediction.
• The benchmark includes metadata such as shortest path length, geodesic distance, and valid viewpoints for evaluation.

Experimental results

Research questions

• RQ1 How does decoupling image goal specification from the agent’s sensor and embodiment improve real-world applicability of visual navigation?
• RQ2 To what extent do focused, instance-specific goal images reduce ambiguity compared to randomly sampled image goals?
• RQ3 What is the performance ceiling of current end-to-end RL methods on instance-specific image goal navigation in realistic, unstructured environments?
• RQ4 How does the proposed benchmark enable fair and standardized evaluation across different navigation methods?
• RQ5 What are the key challenges in generalizing policies trained on instance-specific image goals to unseen environments?

Key findings

• The proposed InstanceImageNav benchmark uses goal images focused on specific object instances, significantly reducing ambiguity compared to randomly sampled image goals.
• Goal images are captured with camera parameters independent of the agent’s sensors, increasing flexibility and realism for user-driven applications.
• The benchmark is instantiated in the Habitat-Sim environment using HM3D scenes with semantic annotations, enabling realistic and diverse navigation tasks.
• A model-free RL baseline achieves only 5.5% success rate and 2.3% SPL on the validation split, indicating substantial room for improvement.
• The training split contains 7,032,000 episodes and the validation split 2,340, providing a large-scale dataset for evaluation and research.
• The release of a public leaderboard on EvalAI is expected to standardize evaluation and accelerate progress in instance-specific visual navigation.