[Paper Review] Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning

Yuke Zhu, Roozbeh Mottaghi+5 more|arXiv (Cornell University)|Sep 16, 2016

Reinforcement Learning in Robotics52 references163 citations

TL;DR

The paper presents a target-driven deep reinforcement learning model with a Siamese actor-critic architecture and the AI2-THOR simulation framework, enabling generalization across targets and scenes with improved data efficiency and sim-to-real transfer.

ABSTRACT

Two less addressed issues of deep reinforcement learning are (1) lack of generalization capability to new target goals, and (2) data inefficiency i.e., the model requires several (and often costly) episodes of trial and error to converge, which makes it impractical to be applied to real-world scenarios. In this paper, we address these two issues and apply our model to the task of target-driven visual navigation. To address the first issue, we propose an actor-critic model whose policy is a function of the goal as well as the current state, which allows to better generalize. To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine. Our framework enables agents to take actions and interact with objects. Hence, we can collect a huge number of training samples efficiently. We show that our proposed method (1) converges faster than the state-of-the-art deep reinforcement learning methods, (2) generalizes across targets and across scenes, (3) generalizes to a real robot scenario with a small amount of fine-tuning (although the model is trained in simulation), (4) is end-to-end trainable and does not need feature engineering, feature matching between frames or 3D reconstruction of the environment. The supplementary video can be accessed at the following link: https://youtu.be/SmBxMDiOrvs.

Motivation & Objective

• Address generalization gaps in deep reinforcement learning for visual navigation by incorporating the target into the policy input.
• Develop a high-quality simulation environment (AI2-THOR) to enable scalable data collection and realistic indoor interactions.
• Propose a target-driven policy that generalizes across targets without re-training.
• Demonstrate end-to-end trainability without feature engineering or explicit 3D reconstruction.
• Evaluate generalization to new targets, new scenes, continuous space, and real-robot transfer.

Proposed method

• Propose a deep siamese actor-critic network that processes current observation and target images in parallel with weight-sharing, producing a joint embedding for policy and value outputs.
• Use scene-specific final layers to capture layout-specific navigation cues while sharing generic siamese layers across targets and scenes.
• Discretize actions to move forward/backward and turn left/right with Gaussian noise to model dynamics.
• Fix ImageNet-pretrained ResNet-50 backbones as feature extractors, stack 4 history frames as input, and project embeddings to a 512-d space.
• Train with an A3C-like asynchronous protocol where each thread targets a different navigation goal, updating scene-specific and generic layers accordingly.
• Reward design includes a sparse goal-reaching reward (10.0) with a small time penalty (-0.01) to encourage shorter trajectories.

Experimental results

Research questions

• RQ1 Can a target-driven policy generalize to unseen targets within the same scene?
• RQ2 Can the model generalize to targets in unseen scenes while reusing learned representations?
• RQ3 Does sharing information across targets improve data efficiency compared to traditional DRL baselines?
• RQ4 Is the approach transferable to continuous spaces and real-robot scenarios with limited fine-tuning?

Key findings

• The final target-driven model achieves substantially shorter average trajectories (210.7 steps) than baselines including A3C variants and a single-branch target model.
• Data efficiency improves, with the final model outperforming state-of-the-art DRL methods after 100M training frames.
• The model generalizes across unseen targets within a scene and across unseen scenes, aided by shared siamese layers and scene-specific layers.
• t-SNE visualizations indicate embedding space preserves spatial layout, suggesting implicit localization/mapping.
• In continuous-space tasks, the model reaches doors/targets with significantly fewer steps than random behavior, albeit with more training frames.
• Robot experiments show successful sim-to-real transfer with small fine-tuning, and transferring learned generic layers accelerates convergence.