[Paper Review] Rearrangement: A Challenge for Embodied AI

Dhruv Batra, Anne Lynn S. Chang+10 more|arXiv (Cornell University)|Nov 3, 2020

Robot Manipulation and LearningEngineering65 references101 citations

TL;DR

The paper proposes rearrangement as a canonical task for Embodied AI, defines a formal framework, and provides four simulation testbeds to standardize evaluation across platforms.

ABSTRACT

We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. A standard task can focus the development of new techniques and serve as a source of trained models that can be transferred to other settings. In the rearrangement task, the goal is to bring a given physical environment into a specified state. The goal state can be specified by object poses, by images, by a description in language, or by letting the agent experience the environment in the goal state. We characterize rearrangement scenarios along different axes and describe metrics for benchmarking rearrangement performance. To facilitate research and exploration, we present experimental testbeds of rearrangement scenarios in four different simulation environments. We anticipate that other datasets will be released and new simulation platforms will be built to support training of rearrangement agents and their deployment on physical systems.

Motivation & Objective

• Propose rearrangement as a standardized, canonical task to unify Embodied AI research.
• Define an end-to-end evaluation protocol that handles diverse goal specifications (geometric, image, language, experience, predicates).
• Characterize rearrangement across embodiment, perception, and manipulation axes.
• Provide experimental testbeds in multiple simulation environments to foster cross-platform research and model transfer.
• Encourage strong generalization and realistic sensing in evaluation and deployment.

Proposed method

• Formalize rearrangement within a POMDP-like framework for rigid and articulated objects.
• Describe goal specification mechanisms: GeometricGoal, ImageGoal, LanguageGoal, ExperienceGoal, PredicateGoal.
• Survey embodiment options from abstract magic-pointer to full physical simulation and sensor modalities.
• Propose evaluation that scores episodes on a 0–1 scale and emphasizes end-to-end perception-to-action pipelines.
• Release rearrangement scenarios in THOR, RLBench, SAPIEN, and Habitat to enable cross-platform experimentation.

Experimental results

Research questions

• RQ1 What is a general, end-to-end definition of rearrangement in embodied contexts?
• RQ2 How can diverse goal specifications be unified under a single evaluation protocol?
• RQ3 How does embodiment and sensor choice affect rearrangement performance and progress toward embodied AI?
• RQ4 What are effective testbeds and benchmarks to enable cross-platform comparison and transfer to physical systems?
• RQ5 How should generalization be defined and measured in rearrangement tasks?

Key findings

• Rearrangement is defined as transforming an environment from an initial state to a goal state under partial observations with a scored 0–1 episode reward.
• The framework accommodates multiple goal specifications including geometric, visual, language, experience, and predicates.
• Four benchmark simulators are released (THOR, RLBench, SAPIEN, Habitat) to support end-to-end evaluation across platforms.
• A spectrum of embodiment choices is discussed, from abstract pointers to full physical simulation, with guidance on when to use each.
• The authors advocate strong generalization, evaluating agents on unseen objects and environments with realistic sensing and no privileged information.