[Paper Review] On Bringing Robots Home

Nur Muhammad Mahi Shafiullah, Anant Rai+5 more|arXiv (Cornell University)|Nov 27, 2023

Social Robot Interaction and HRI4 citations

TL;DR

This paper introduces Dobb·E, a low-cost, general-purpose robotic system that enables home robots to learn new tasks through just five minutes of human demonstration using a DIY 'Stick' tool. Trained on a 13-hour dataset from 22 NYC homes, Dobb·E achieves 81% success rate across 109 household tasks using Home Pretrained Representations (HPR), with full code, models, and hardware designs open-sourced to accelerate home robotics research.

ABSTRACT

Throughout history, we have successfully integrated various machines into our homes. Dishwashers, laundry machines, stand mixers, and robot vacuums are a few recent examples. However, these machines excel at performing only a single task effectively. The concept of a "generalist machine" in homes - a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective - has long been a goal in robotics that has been steadily pursued for decades. In this work, we initiate a large-scale effort towards this goal by introducing Dobb-E, an affordable yet versatile general-purpose system for learning robotic manipulation within household settings. Dobb-E can learn a new task with only five minutes of a user showing it how to do it, thanks to a demonstration collection tool ("The Stick") we built out of cheap parts and iPhones. We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR). Then, in a novel home environment, with five minutes of demonstrations and fifteen minutes of adapting the HPR model, we show that Dobb-E can reliably solve the task on the Stretch, a mobile robot readily available on the market. Across roughly 30 days of experimentation in homes of New York City and surrounding areas, we test our system in 10 homes, with a total of 109 tasks in different environments, and finally achieve a success rate of 81%. Beyond success percentages, our experiments reveal a plethora of unique challenges absent or ignored in lab robotics. These range from effects of strong shadows, to variable demonstration quality by non-expert users. With the hope of accelerating research on home robots, and eventually seeing robot butlers in every home, we open-source Dobb-E software stack and models, our data, and our hardware designs at https://dobb-e.com

Motivation & Objective

• To develop a cost-effective, general-purpose robotic system capable of learning diverse household tasks in real-world homes.
• To address the gap between lab-based robotics performance and real-world home deployment by collecting data in uncontrolled, diverse home environments.
• To enable rapid task learning through imitation learning using minimal human demonstrations, enhancing user comfort and safety.
• To establish a scalable framework for home robotics by open-sourcing hardware, software, datasets, and pretrained models.
• To identify and document real-world challenges—such as lighting variations and sensor limitations—overlooked in controlled lab settings.

Proposed method

• The 'Stick' is a low-cost demonstration tool made from a reacher-grabber, 3D-printed parts, and an iPhone, enabling non-expert users to record task demonstrations in unfamiliar homes.
• 13 hours of human demonstration data were collected across 22 NYC homes, forming the Homes of New York (HoNY) dataset with 5,620 demonstrations across 216 environments.
• Home Pretrained Representations (HPR) are trained using self-supervised learning (SSL) on RGB and depth video data from the HoNY dataset to learn robust visual and spatial representations.
• For new tasks, the HPR model is fine-tuned using just 24 demonstrations (5 minutes of data) via behavior cloning, enabling rapid adaptation to new environments.
• The system is deployed on the commercially available Hello Robot Stretch robot, with direct data transfer from the Stick to the robot via synchronized iPhone mounts.
• A holistic pipeline integrates hardware, data collection, pretraining, and deployment, with all components open-sourced for reproducibility and collaboration.

Figure 1 : We present Dobb·E, a simple framework to train robots, which is then field tested in homes across New York City. In under 30 mins of training per task, Dobb·E achieves 81% success rates on simple household tasks.

Experimental results

Research questions

• RQ1 Can a low-cost, general-purpose robot learn diverse household tasks using only five minutes of human demonstration in real homes?
• RQ2 How does pretraining on a large-scale, real-world home dataset improve zero-shot and few-shot generalization in robotic manipulation?
• RQ3 What are the dominant real-world failure modes—such as lighting, sensor noise, or hardware limitations—that hinder performance in unstructured home environments?
• RQ4 To what extent does demonstrator expertise or data quality affect downstream task success rates in real home deployments?
• RQ5 How can open-source hardware, software, and datasets accelerate research in home robotics and reduce barriers to entry?

Key findings

• Dobb·E achieved an 81% success rate across 109 household tasks in 10 real homes, with a median of 24 demonstrations per task and 15 minutes of fine-tuning.
• The system successfully generalized to new environments and tasks without domain adaptation, demonstrating the effectiveness of Home Pretrained Representations (HPR).
• Lighting variations and strong shadows were identified as major failure factors, with 15% of failures directly traceable to poor illumination conditions.
• Demonstrations collected by non-expert users led to variable quality, with 22% of tasks requiring re-collection due to unclear or inconsistent actions.
• Sensor limitations, particularly in depth perception and occlusion handling, contributed to 18% of failures, especially in cluttered or complex scenes.
• The open-sourcing of the full stack—including hardware designs, code, dataset (77 GB with RGB and depth), and pretrained models—enables full reproducibility and community extension.

Figure 2 : (A) We design a new imitation learning framework, starting with a data collection tool. (B) Using this data collection tool, users can easily collect demonstrations for household tasks. (C) Using a similar setup on a robot, (D) we can transfer those demos using behavior cloning techniques