[Paper Review] BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

Sanjana Srivastava, Chengshu Li+12 more|arXiv (Cornell University)|Aug 6, 2021

Reinforcement Learning in RoboticsComputer Science57 references36 citations

TL;DR

BEHAVIOR proposes a 100-task benchmark for embodied AI in realistic, diverse, long-horizon household activities, with a predicate-logic domain language, simulator-agnostic requirements, and human VR demonstrations for evaluation.

ABSTRACT

We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation. These activities are designed to be realistic, diverse, and complex, aiming to reproduce the challenges that agents must face in the real world. Building such a benchmark poses three fundamental difficulties for each activity: definition (it can differ by time, place, or person), instantiation in a simulator, and evaluation. BEHAVIOR addresses these with three innovations. First, we propose an object-centric, predicate logic-based description language for expressing an activity's initial and goal conditions, enabling generation of diverse instances for any activity. Second, we identify the simulator-agnostic features required by an underlying environment to support BEHAVIOR, and demonstrate its realization in one such simulator. Third, we introduce a set of metrics to measure task progress and efficiency, absolute and relative to human demonstrators. We include 500 human demonstrations in virtual reality (VR) to serve as the human ground truth. Our experiments demonstrate that even state of the art embodied AI solutions struggle with the level of realism, diversity, and complexity imposed by the activities in our benchmark. We make BEHAVIOR publicly available at behavior.stanford.edu to facilitate and calibrate the development of new embodied AI solutions.

Motivation & Objective

• Define a realistic, diverse, and complex set of 100 everyday household activities grounded in real-world data.
• Provide a predicate-logic-based domain definition language (BDDL) to express initial and goal conditions for unlimited instantiations.
• Identify simulator-agnostic requirements and provide a functional instantiation in iGibson 2.0.
• Establish robust evaluation metrics including a human-relative success and efficiency framework.
• Offer a VR-based human demonstration dataset (500 demos) as ground truth for evaluation and imitation learning.

Proposed method

• Introduce BEHAVIOR Domain Definition Language (BDDL), a predicate-logic representation mapping simulated states to semantic predicates for initial and goal conditions.
• Propose environment-agnostic functional requirements for realistic simulation and instantiate BEHAVIOR in iGibson 2.0 with 1217 object models across 391 categories.
• Define activities as POMDPs with a formal tau = {S_tau,0, S_tau,g} and provide mechanisms to sample infinite valid initial states and solutions via BDDL.
• Develop a comprehensive set of evaluation metrics, including a primary success score Q and six secondary efficiency metrics, normalized by human performance using a VR demonstation dataset.
• Provide 500 VR demonstrations (758.5 minutes) as human ground truth and support analysis for imitation learning and human benchmarking.

Experimental results

Research questions

• RQ1 How to define, instantiate, and evaluate realistic, long-horizon household activities in embodied AI benchmarks?
• RQ2 Can a predicate-logic domain language (BDDL) enable infinite, diverse instantiations of activities across scenes?
• RQ3 What simulator requirements are necessary to realize BEHAVIOR, and how well do current embodied AI methods perform on BEHAVIOR tasks?
• RQ4 How do human VR demonstrations compare to AI agents on realism, diversity, and complexity metrics?

Key findings

• BEHAVIOR yields 100 activities with long horizons, multiple objects, and varied state changes that are challenging for state-of-the-art RL methods.
• A human VR dataset of 500 demonstrations provides a rich ground truth for evaluation and imitation learning.
• RL agents struggle to achieve any goal predicates on the original BEHAVIOR activities even with advanced sensing and actions; reducing horizon or simplifying physics improves performance.
• Diversity and complexity across scenes, objects, and initial states significantly degrade RL performance, highlighting the need for hierarchical or planning-based approaches.