[Paper Review] Unified Human-Scene Interaction via Prompted Chain-of-Contacts

Zeqi Xiao, Tai Wang+6 more|arXiv (Cornell University)|Sep 14, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes UniHSI, a unified human-scene interaction framework that enables versatile, long-horizon physical interactions via natural language commands. By modeling interactions as Chain of Contacts—sequential human joint-object part contact pairs—UniHSI uses an LLM-based planner to generate task plans and a unified controller to execute them, achieving annotation-free training and strong generalization on real scanned scenes.

ABSTRACT

Human-Scene Interaction (HSI) is a vital component of fields like embodied AI and virtual reality. Despite advancements in motion quality and physical plausibility, two pivotal factors, versatile interaction control and the development of a user-friendly interface, require further exploration before the practical application of HSI. This paper presents a unified HSI framework, UniHSI, which supports unified control of diverse interactions through language commands. This framework is built upon the definition of interaction as Chain of Contacts (CoC): steps of human joint-object part pairs, which is inspired by the strong correlation between interaction types and human-object contact regions. Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and a Unified Controller that turns CoC into uniform task execution. To facilitate training and evaluation, we collect a new dataset named ScenePlan that encompasses thousands of task plans generated by LLMs based on diverse scenarios. Comprehensive experiments demonstrate the effectiveness of our framework in versatile task execution and generalizability to real scanned scenes. The project page is at https://github.com/OpenRobotLab/UniHSI .

Motivation & Objective

• Address the lack of versatile interaction control and user-friendly interfaces in human-scene interaction (HSI) systems.
• Overcome limitations in existing methods that support only narrow interaction types or require extensive human-annotated motion data.
• Enable long-horizon, multi-round transitions in HSI through a unified, scalable framework.
• Develop a system that supports fine-grained, multi-object interactions using only language input and scene semantics.
• Reduce reliance on costly human-annotated interaction datasets by leveraging LLMs to generate interaction plans.

Proposed method

• Define human-scene interaction as a Chain of Contacts (CoC): sequential pairs of human joints and object parts in contact, capturing interaction dynamics.
• Use a Large Language Model (LLM) Planner to convert natural language commands into structured CoC task plans using prompt engineering.
• Design a Unified Controller with a TaskParser that processes joint poses and object point clouds to generate uniform task observations and objectives.
• Integrate adversarial motion prior and physical simulation to ensure realistic and physically plausible motion generation.
• Train the system end-to-end without interaction annotations by leveraging LLM-generated interaction plans.
• Construct the ScenePlan dataset with thousands of LLM-generated interaction plans from PartNet and ScanNet-based scenes for training and evaluation.

Experimental results

Research questions

• RQ1 Can a unified framework support diverse human-scene interactions using only natural language commands?
• RQ2 Can the Chain of Contacts (CoC) formulation generalize across multiple interaction types, including multi-round and multi-object interactions?
• RQ3 Can LLMs effectively generate accurate and executable interaction plans without human-annotated data?
• RQ4 How well does the unified controller generalize to real scanned 3D scenes beyond synthetic training environments?
• RQ5 To what extent does the framework enable fine-grained, long-horizon control with precise joint and object part coordination?

Key findings

• UniHSI achieves unified, long-horizon control of diverse interactions using only language commands, including complex transitions like sitting with hands on arms or lying on a bed with knees up.
• The framework supports fine-grained control over 15 whole-body joints and arbitrary object parts, enabling multi-object and multi-contact interactions.
• The ScenePlan dataset contains over 1,000 unique interaction plans across two versions, with detailed contact formations for complex behaviors like lying on a bed or sitting with elbows on arms.
• Experiments show strong generalization to real scanned scenes from ScanNet, demonstrating robustness beyond synthetic training data.
• The system is interaction annotation-free, relying solely on LLM-generated plans, significantly reducing data collection costs compared to prior methods.
• Compared to prior SOTA methods, UniHSI uniquely supports unified interaction, language input, long-horizon transitions, annotation-free training, and full-body joint control.