[Paper Review] Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

Xavier Puig, Tianmin Shu+6 more|arXiv (Cornell University)|Oct 19, 2020

Reinforcement Learning in Robotics54 references40 citations

TL;DR

WAH introduces a two-stage challenge where an AI watches a human-like agent to infer goals and then collaborates in a new environment to complete household tasks as fast as possible. It evaluates social perception and human-AI collaboration in VirtualHome-Social.

ABSTRACT

In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration). For this challenge, we build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines. We evaluate the performance of AI agents with the human-like agent as well as with real humans using objective metrics and subjective user ratings. Experimental results demonstrate that the proposed challenge and virtual environment enable a systematic evaluation on the important aspects of machine social intelligence at scale.

Motivation & Objective

• Motivate and define a new AI challenge to test social perception (goal inference from observation) and collaborative planning (efficiently helping in unseen environments).
• Extend VirtualHome to a multi-agent setting, enabling interactions between AI agents, built-in human-like agents, and real humans for complex household tasks.
• Provide a benchmark with goal-inference models and planning/DRL baselines to study machine social intelligence at scale.

Proposed method

• Propose a two-stage Watch-And-Help framework: Watch stage where Bob observes Alice performing a task to infer her goal; Help stage where Bob collaborates with Alice to achieve the same goal efficiently in a different environment.
• Build VirtualHome-Social, a multi-agent household environment with symbolic and visual observations, object interactions, and built-in human-like agents that can replan in response to Bob.
• Define tasks by a goal described as a set of predicates with counts, representing five predicate sets of household activities (e.g., setting a dinner table, fridge groceries, cooking, washing dishes, reading with snacks).
• Implement a goal inference model that uses a Transformer+LSTM over visible objects to predict predicate counts; use a hierarchical planning/learning architecture where high-level subgoals are selected and low-level policies/planners execute actions.
• Evaluate baselines including HP (hierarchical planner with MCTS RP), Hybrid (RL+RP), HRL (hierarchical RL), and Random, plus Oracle variants with full goal knowledge.

Experimental results

Research questions

• RQ1 Can an AI infer human goals from a single demonstration in a realistic multi-agent environment?
• RQ2 How well can an AI agent plan and coordinate with a human-like agent to minimize task completion time in unseen environments?
• RQ3 What are the key factors that enable effective human-AI collaboration in household tasks?
• RQ4 How does goal inference accuracy impact collaborative performance across different task categories?

Key findings

• Goal inference precision/recall on test-1 achieve 0.85/0.96; using full demonstrations yields higher precision/recall (0.99/0.99).
• HP baseline with true goals achieves best performance among non-oracle baselines in the Help stage, with higher success and speedup than other baselines.
• Using inferred goals (RG) can hurt performance due to conflicts with Alice’s plan, highlighting the need for dynamic goal adjustment.
• Test-2 (multi-activity tasks) reduces goal-prediction accuracy (0.68/0.64) and degrades helping performance, showing generalization challenges to multi-activity scenarios.
• Alice alone achieves 95.4% success in the Help stage, while HP baseline reaches 88.6% with 0.21 speedup; performance drops in test-2 due to poorer goal recognition.
• Human experiments show AI baselines preserve relative ranking when paired with real humans, and user ratings align with objective metrics.