[Paper Review] iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks

Chengshu Li, Fei Xia+14 more|arXiv (Cornell University)|Aug 6, 2021

Robot Manipulation and Learning33 references62 citations

TL;DR

iGibson 2.0 extends the Gibson simulation with object-centric extended states (temperature, wetness, cleanliness, toggled and sliced), predicate-based logic and sampling, plus a VR interface for collecting demonstrations, enabling six new household tasks and imitation learning.

ABSTRACT

Recent research in embodied AI has been boosted by the use of simulation environments to develop and train robot learning approaches. However, the use of simulation has skewed the attention to tasks that only require what robotics simulators can simulate: motion and physical contact. We present iGibson 2.0, an open-source simulation environment that supports the simulation of a more diverse set of household tasks through three key innovations. First, iGibson 2.0 supports object states, including temperature, wetness level, cleanliness level, and toggled and sliced states, necessary to cover a wider range of tasks. Second, iGibson 2.0 implements a set of predicate logic functions that map the simulator states to logic states like Cooked or Soaked. Additionally, given a logic state, iGibson 2.0 can sample valid physical states that satisfy it. This functionality can generate potentially infinite instances of tasks with minimal effort from the users. The sampling mechanism allows our scenes to be more densely populated with small objects in semantically meaningful locations. Third, iGibson 2.0 includes a virtual reality (VR) interface to immerse humans in its scenes to collect demonstrations. As a result, we can collect demonstrations from humans on these new types of tasks, and use them for imitation learning. We evaluate the new capabilities of iGibson 2.0 to enable robot learning of novel tasks, in the hope of demonstrating the potential of this new simulator to support new research in embodied AI. iGibson 2.0 and its new dataset are publicly available at http://svl.stanford.edu/igibson/.

Motivation & Objective

• Motivate extending robot simulation beyond kinodynamics to cover object state changes relevant to household tasks (e.g., temperature, wetness, cleanliness).
• Introduce a logical-predicate and generative sampling framework to map physical states to semantic states and to seed diverse task instances.
• Provide a VR interface to collect human demonstrations for imitation learning on new tasks.
• Demonstrate the ability to populate densely populated scenes with semantically-placed small objects to support realistic task learning.
• Evaluate six novel tasks to show feasibility of learning with extended states and VR-guided demonstrations.

Proposed method

• Introduce extended object states: temperature, wetness level, cleanliness level, toggled state, and sliced state maintained per object category.
• Define unary and binary logical predicates (e.g., Cooked, Soaked, InsideOf) that map physical states to semantic states and enable language-grounded representation.
• Implement a generative sampling system to create valid simulated states from logical states, enabling semantically meaningful scene initialization.
• Develop a rule-based mechanism to densely populate scenes using the generative system, increasing realism without manual authoring.
• Provide a VR interface (OpenVR-compatible) for immersive demonstrations and introduce assistive grasping to enable natural bimanual manipulation in VR.
• Support deterministically replayable logs of states for post-hoc sensor signal generation and imitation-learning workflows.

Experimental results

Research questions

• RQ1 Can extended object states enable learning of new household tasks not feasible in prior simulators?
• RQ2 How effective is the predicate-based generative system at creating diverse, realistic task initializations?
• RQ3 Does a VR-based demonstration interface improve imitation learning for bimanual manipulation in these extended-state tasks?
• RQ4 What are the breaking points of reinforcement and imitation learning on tasks involving temperature, wetness, and slicing?

Key findings

• Six novel tasks requiring extended states were evaluated: Grasping Book, Soaking Towel, Cleaning Stained Shelf, Cooking Meat, Slicing Fruit, and Bimanual Pick and Place.
• With simplified grasping, SAC achieved 100% success on Grasping Book, Soaking Towel, Cleaning Stained Shelf, and Cooking Meat; Slicing Fruit reached 15% (bimanual humanoid) and 0% (Fetch).
• Bimanual Pick and Place with the humanoid model achieved 0% due to coordination challenges; Fetch with full-grasping complexity achieved 25% on two tasks and 0% on three others.
• Imitation learning with VR demonstrations for the bimanual task showed 19% and 46% success when starting 6s and 3s before goal, respectively.
• Ablation on Soaking Towel demonstrated robustness gains with increasing variability: 19%, 79%, and 87% success for progressively diverse initial states.
• The VR interface enables collecting demonstrations for imitation learning and highlights covariate-shift challenges in long, multi-step tasks.