[Paper Review] The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation

Kaichun Mo, Haoxiang Li+2 more|arXiv (Cornell University)|Feb 24, 2018

Multimodal Machine Learning ApplicationsComputer Science26 references22 citations

TL;DR

This paper introduces the AdobeIndoorNav dataset, a high-quality, real-world indoor visual navigation benchmark with densely sampled 360° panoramic images and 3D reconstructions across 24 scenes. It enables deep reinforcement learning (DRL) training with realistic visual inputs, and demonstrates that standard DRL policies fail to generalize to nearby targets due to poor spatial feature representation, which is mitigated by using spatial-aware features from early CNN layers, improving zero-shot generalization accuracy from 53.0% to 72.6% on unseen scenes.

ABSTRACT

Deep reinforcement learning (DRL) demonstrates its potential in learning a model-free navigation policy for robot visual navigation. However, the data-demanding algorithm relies on a large number of navigation trajectories in training. Existing datasets supporting training such robot navigation algorithms consist of either 3D synthetic scenes or reconstructed scenes. Synthetic data suffers from domain gap to the real-world scenes while visual inputs rendered from 3D reconstructed scenes have undesired holes and artifacts. In this paper, we present a new dataset collected in real-world to facilitate the research in DRL based visual navigation. Our dataset includes 3D reconstruction for real-world scenes as well as densely captured real 2D images from the scenes. It provides high-quality visual inputs with real-world scene complexity to the robot at dense grid locations. We further study and benchmark one recent DRL based navigation algorithm and present our attempts and thoughts on improving its generalizability to unseen test targets in the scenes.

Motivation & Objective

• To address the domain gap between synthetic and real-world scenes in DRL-based robot navigation by providing a high-fidelity real-world dataset.
• To support training of DRL policies using realistic, densely sampled visual inputs from actual indoor environments.
• To investigate the generalization failure of state-of-the-art DRL navigation policies in real-world settings, particularly near-target navigation errors.
• To propose and empirically validate a spatial-aware feature representation that improves policy robustness to similar nearby views.

Proposed method

• Collects 360° panoramic images and 3D point cloud reconstructions at dense grid locations (0.4–0.5 m spacing) across 24 real indoor scenes using a semi-automatic, re-producible pipeline.
• Uses a real robot with SLAM to capture high-quality visual inputs and reconstruct 3D scenes, minimizing artifacts common in 3D reconstruction pipelines.
• Adopts a target-driven DRL framework where the agent learns to navigate from visual input to a target view using four discrete actions (forward, backward, left, right).
• Replaces standard 2,048-dim global average pooled CNN features with spatial-aware $7\times7\times2048$ feature maps from earlier convolutional layers to preserve spatial context.
• Employs a siamese network to evaluate feature quality by predicting the correct action between two nearby views (e.g., forward/backward, left/right).
• Trains and tests the siamese model on both same-scene and cross-scene image pairs to assess zero-shot generalization performance.

Experimental results

Research questions

• RQ1 Can a DRL-based visual navigation policy trained on real-world visual inputs generalize to unseen test targets in the same environment?
• RQ2 Why do DRL policies fail to reach nearby targets despite learning coarse layout awareness?
• RQ3 Does preserving spatial structure in visual features improve generalization to similar nearby views in real-world navigation?
• RQ4 How does the performance of a DRL policy change when using spatial-aware features versus standard global features?
• RQ5 Can a diagnostic siamese network effectively measure the discriminative power of visual features for near-target navigation?

Key findings

• The standard DRL policy fails to reach nearby targets due to inability to distinguish visually similar views, even after successful coarse navigation to the target vicinity.
• Using 2,048-dim global average pooled features results in only 83.5% accuracy in matching nearby views on the same scene.
• Replacing global features with spatial-aware $7\times7\times2048$ feature maps improves matching accuracy to 87.5% on same scenes.
• On unseen scenes, the improvement is even more significant: zero-shot generalization accuracy increases from 53.0% to 72.6% with spatial-aware features.
• The diagnostic experiment confirms that spatial-aware features encode more discriminative spatial information critical for resolving near-target navigation failures.