[Paper Review] Building Generalizable Agents with a Realistic and Rich 3D Environment

Yi Wu, Yuxin Wu+2 more|arXiv (Cornell University)|Jan 7, 2018

Multimodal Machine Learning ApplicationsComputer Science253 citations

TL;DR

The paper introduces House3D, a large-scale, realistic 3D indoor environment built from SUNCG to train/benchmark semantic-navigation agents, and demonstrates that multi-level data augmentation yields better generalization to unseen houses."

ABSTRACT

Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et.al.). The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel- & task-level augmentations such as domain randomization (Toubin et. al.) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at http://github.com/facebookresearch/House3D.

Motivation & Objective

• Motivate the need for agents that generalize across unseen 3D environments with diverse rooms and objects.
• Create a large, diverse, label-rich environment (House3D) derived from SUNCG to study semantic navigation.
• Develop RoomNav as a concept-driven navigation benchmark to test generalization.
• Propose gated-attention policy architectures to handle multi-target conditioning in navigation tasks.
• Evaluate how pixel-, task-, and scene-level augmentations improve generalization to unseen environments.

Proposed method

• Construct House3D from SUNCG, providing 45,622 scenes with 80+ object categories and multi-modal observations (RGB, depth, segmentation, top-down maps).
• Define RoomNav: navigate to a target semantic concept (e.g., kitchen) given as an instruction I, evaluating in unseen houses.
• Use gated-CNN (DDPG) and gated-LSTM (A3C) policies to condition on target concepts via a gating mechanism for feature fusion.
• Apply pixel-level (domain randomization), task-level (auxiliary targets), and scene-level (larger environment sets) augmentations to enhance generalization.
• Compare RGB-only, RGB+Depth, and Mask+Depth inputs, showing depth/semantic inputs improve robustness and generalization.
• Train with standard RL algorithms (DDPG, A3C) and use reward shaping to guide learning during training.”],
• research_questions:[
• What is the impact of different input modalities (RGB, Depth, semantic masks) on generalization to unseen 3D environments?
• How do pixel-, task-, and scene-level augmentations individually and jointly affect generalization in RoomNav?
• Can gated-attention policies effectively condition on high-level semantic targets for multi-target navigation in diverse houses?
• Does increasing environmental diversity (scene-level augmentation) reduce overfitting and improve success rates in unseen environments?
• What is the performance gain of using depth/semantic signals over RGB-only inputs in RoomNav?

Experimental results

Research questions

• RQ1 Does the gated-attention architecture with semantic inputs outperform baselines in unseen environments?
• RQ2 How do discrete-action (A3C) vs continuous-action (DDPG) setups compare under various input modalities?
• RQ3 What is the best combination of augmentation techniques for RoomNav generalization?

Key findings

• A final gated-LSTM agent with semantic signals achieves 35.8% success on 50 unseen environments, about 10% higher than the 25.7% baseline.
• Depth and semantic segmentation inputs improve generalization over RGB-only inputs.
• Pixel-level augmentation via domain randomization yields improvements comparable to segmentation masks on smaller datasets.
• Task-level augmentation with auxiliary targets reduces overfitting and enhances generalization for RGB inputs.
• Scene-level augmentation by training on a larger, more diverse set of houses significantly reduces overfitting and improves test performance.
• Across inputs, LSTM-based policies outperform CNN-based ones on larger, more complex training sets, indicating benefits of memory for generalization.