[Paper Review] On the Strengths and Weaknesses of Data for Open-set Embodied Assistance

Pradyumna Tambwekar, Andrew Silva+4 more|arXiv (Cornell University)|Mar 5, 2026

Multimodal Machine Learning Applications0 citations

TL;DR

The paper trains a multimodal, instruction-tuned embodied model on synthetic Overcooked trajectories to enable Open-Set Corrective Assistance, showing generalization to unseen defects and novel tasks, and analyzes how diverse assistive data shapes performance.<br/>Compared against GPT-4o baselines, the model demonstrates stronger generalization, with insights on dataset design for embodied assistance.

ABSTRACT

Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive embodied foundation models. In this paper, we investigate the generalization capabilities of a multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called extbf{Open-Set Corrective Assistance}, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach provides key insights into the nature of assistive datasets required to enable open-set assistive intelligence. In particular, we show that performant models benefit from datasets that cover different aspects of assistance, including multimodal grounding, defect inference, and exposure to diverse scenarios.

Motivation & Objective

• Motivate and enable open-set corrective assistance where there is no predefined set of corrections.
• Investigate generalization along defect types and task configurations using synthetic Overcooked data.
• Assess how diverse, multimodal assistive data influences grounding, reasoning, and action generation.

Proposed method

• Fine-tune a LLaMA-3 base with a ViT image encoder to create a multimodal model that outputs either language coaching or corrective actions from trajectory data.
• Generate synthetic long-horizon user trajectories in Overcooked with diverse defect wrappers to cover cognitive planning and visuospatial impairments.
• Create grounding datasets (Image-QA, Trajectory-QA, Video-QA) and task-specific datasets (Coaching, Corrections, Defect Delineation) to train for open-set assistance.
• Use ground-truth corrections generated by predicting the next action of defect-free trajectories and synthetic coaching snippets produced by GPT-4o with diverse personas.
• Evaluate with few-shot fine-tuning (10 examples per new defect) against GPT-4o baselines and across held-out defects and novel recipes.
• Explore ablations to understand the impact of multi-task training and grounding data on generalization.

Experimental results

Research questions

• RQ1 Can an embodied foundation model trained on synthetic assistive data generalize to unseen defective user behaviors (open-set defects) and to novel task configurations (recipes)?
• RQ2 What dataset design characteristics (multimodal grounding, reasoning traces, task decomposition) most promote open-set corrective assistance?
• RQ3 How does model scale (1B vs 8B parameters) influence zero-shot and few-shot generalization in open-set scenarios?
• RQ4 Does joint training on multiple assistive tasks and grounding data improve performance on unseen defects and new tasks?

Key findings

• The proposed model trained on diverse synthetic assistive data outperforms GPT-4o baselines in both coaching and correction tasks on held-out defects in both zero-shot and few-shot settings.
• Across held-out defects, 1B and 8B variants achieve 76.60 and 77.80 coaching scores, and 55.70 and 54.60 corrections respectively, outperforming baselines (GPT-4o: 21.00 coaching, 20.40 corrections).
• Reasoning traces can boost coaching in some settings but may cause mode-collapse; zero-shot with reasoning yields mixed gains and sometimes degrades coaching performance vs. non-reasoned inputs.
• Task generalization to new recipes improves with larger models, indicating stronger multimodal grounding is needed for compositionality in unseen tasks.
• Joint training on coaching, corrections, and defect delineation generally improves downstream assistive performance compared to single-task training; grounding data (especially Trajectory-QA) aids generalization to novel configurations.
• Co-training with grounding datasets improves visual compositionality, aiding generalization to new task configurations (DT-most effective among grounding datasets).