[Paper Review] CHALET: Cornell House Agent Learning Environment

Claudia Yan, Dipendra Misra+4 more|arXiv (Cornell University)|Jan 23, 2018

Reinforcement Learning in Robotics19 references71 citations

TL;DR

CHALET is a 3D Unity-based house simulator supporting navigation and manipulation to train agents in language, vision, and planning within 58 rooms across 10 houses, featuring movable objects, containers, and realistic physics.

ABSTRACT

We present CHALET, a 3D house simulator with support for navigation and manipulation. CHALET includes 58 rooms and 10 house configuration, and allows to easily create new house and room layouts. CHALET supports a range of common household activities, including moving objects, toggling appliances, and placing objects inside closeable containers. The environment and actions available are designed to create a challenging domain to train and evaluate autonomous agents, including for tasks that combine language, vision, and planning in a dynamic environment.

Motivation & Objective

• Provide a rich interactive house environment for training autonomous agents in navigation and manipulation tasks.
• Support a variety of object interactions including moving, opening/closing containers, and placing items.
• Facilitate language grounding and instruction following in a partially observable, dynamic setting.
• Enable scalable generation of diverse house layouts and object configurations for robust learning.

Proposed method

• Implemented CHALET in Unity 3D with C# scripting for cross-platform deployment (including WebGL).
• Supports standalone, simulator, and client modes for varied interaction and ML integration.
• Models continuous agent actions in a first-person perspective with discrete precision options.
• Includes 58 rooms, 10 houses, 150 object types, and 71 manipulable types to enable rich manipulation scenarios.
• Defines a rich action set (move, strafe, look, interact) to enable complex navigation and object interactions.
• Provides evaluation via annotated demonstrations and metrics for navigation error and manipulation accuracy.

Experimental results

Research questions

• RQ1 How can a simulated 3D house environment support complex language-grounded navigation and manipulation tasks?
• RQ2 What are the challenges and requirements for evaluating agent behavior in partially observable, dynamic household settings?
• RQ3 Can a Unity-based simulator provide scalable generation of diverse layouts and object configurations for robust learning?
• RQ4 How well do agents learn to follow instructions involving containers and multi-step manipulation in a realistic environment?

Key findings

• CHALET offers a navigable and manipulable 3D environment with 58 rooms across 10 houses and 150 object types.
• A broad set of manipulable actions enables opening/closing containers and dynamic environment changes.
• The evaluation framework uses demonstrations to compute navigation error and manipulation F1-score against reference actions.
• The environment supports multiple operation modes and easy scenario generation for learning and crowdsourcing.
• CHALET emphasizes language and planning challenges through spatial relations and object state changes that require exploration and memory.