[Paper Review] AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

Jiafei Duan, Wilbert Pumacay+8 more|arXiv (Cornell University)|Oct 1, 2024

Robot Manipulation and Learning4 citations

TL;DR

This paper introduces Aha, an open-source vision-language model that detects and reasons about failures in robotic manipulation through free-form natural language explanations. Trained on the Aha dataset—generated via FailGen, a procedural data augmentation pipeline—Aha outperforms GPT-4o by 10.3% on average and boosts task success rates by 21.4% across three robotic frameworks by enabling adaptive failure feedback in reinforcement learning, planning, and zero-shot trajectory generation.

ABSTRACT

Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, AHA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AHA using FailGen, a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the AHA dataset, AHA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-art VLMs by 35.3% across multiple metrics and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA's failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models.

Motivation & Objective

• To enable robots to autonomously detect and reason about failures in open-world manipulation tasks, moving beyond binary failure detection.
• To address the lack of scalable, diverse failure data for training vision-language models in robotics, especially for failure generalization.
• To develop a foundation model that generalizes across unseen robots, tasks, and environments using free-form reasoning instead of rigid classification.
• To integrate failure reasoning into downstream robotic systems such as reinforcement learning, task and motion planning, and zero-shot trajectory generation.
• To demonstrate that natural language-based failure feedback significantly improves policy performance and robustness in real-world and simulated settings.

Proposed method

• Proposes FailGen, a procedural data generation pipeline that perturbs successful demonstration trajectories in simulation to create diverse failure modes across 79 simulated tasks.
• Constructs the Aha dataset with 49K+ image-query pairs of failure trajectories, covering diverse failure types, robot embodiments, and environmental variations.
• Fine-tunes a vision-language model (Aha-13B) on the Aha dataset using instruction tuning to frame failure detection as a free-form reasoning task.
• Integrates Aha into three robotic frameworks: (1) dense reward shaping for reinforcement learning, (2) task and motion planning with feedback loops, and (3) sub-task verification in zero-shot trajectory generation.
• Uses natural language explanations from Aha to refine reward functions, optimize planning, and improve sub-task verification, enabling adaptive error recovery.
• Employs zero-shot evaluation on out-of-distribution datasets and real-world failure data to assess generalization across unseen tasks, robots, and environments.

Experimental results

Research questions

• RQ1 Can a vision-language model trained solely on simulated failure data generalize effectively to real-world robotic failures and unseen tasks?
• RQ2 Does framing failure detection as a free-form reasoning task improve interpretability and adaptability compared to binary classification or fixed-label approaches?
• RQ3 To what extent can Aha’s failure explanations enhance downstream robotic policy performance in reinforcement learning, planning, and zero-shot generation?
• RQ4 How does Aha compare to state-of-the-art VLMs and proprietary models like GPT-4o in failure reasoning across diverse, out-of-distribution settings?
• RQ5 Can Aha’s failure feedback be effectively reused across multiple robotic frameworks to improve success rates and robustness?

Key findings

• Aha outperforms GPT-4o by 10.3% on average across multiple metrics and datasets, demonstrating superior generalization to unseen failure scenarios.
• Aha exceeds the average performance of six compared models—including five SOTA VLMs—by 35.3% across evaluation metrics, highlighting its robustness and generalization capability.
• When integrated into a reinforcement learning framework, Aha improves task success rates by 21.4% compared to GPT-4o, with a 22.34% higher success rate on five benchmark tasks.
• In task and motion planning (PRoC3S), Aha boosts success rates by 36.7% compared to GPT-4o, due to more accurate failure diagnosis and feedback during LLM-based plan refinement.
• In zero-shot trajectory generation (Manipulate-Anything), replacing GPT-4V with Aha-13B increases reasoning accuracy and overall task success by an average of 5% across four RLBench tasks.
• Aha generalizes effectively to real-world failure datasets and diverse robotic systems despite being fine-tuned only on simulated data, confirming the effectiveness of FailGen’s procedural data augmentation.