[Paper Review] Every Mistake Counts in Assembly

Guodong Ding, Fadime Şener+2 more|arXiv (Cornell University)|Jul 31, 2023

Manufacturing Process and OptimizationEngineering3 citations

TL;DR

This paper proposes a knowledge-based system that detects ordering mistakes in assembly procedures by learning spatial and temporal beliefs from real-world action sequences. Using an online, episodic memory framework, it dynamically infers topological relationships and prerequisite constraints, achieving 71.8% F1 on fine-grained mistake detection—significantly outperforming LSTM and TempAgg baselines.

ABSTRACT

One promising use case of AI assistants is to help with complex procedures like cooking, home repair, and assembly tasks. Can we teach the assistant to interject after the user makes a mistake? This paper targets the problem of identifying ordering mistakes in assembly procedures. We propose a system that can detect ordering mistakes by utilizing a learned knowledge base. Our framework constructs a knowledge base with spatial and temporal beliefs based on observed mistakes. Spatial beliefs depict the topological relationship of the assembling components, while temporal beliefs aggregate prerequisite actions as ordering constraints. With an episodic memory design, our algorithm can dynamically update and construct the belief sets as more actions are observed, all in an online fashion. We demonstrate experimentally that our inferred spatial and temporal beliefs are capable of identifying incorrect orderings in real-world action sequences. To construct the spatial beliefs, we collect a new set of coarse-level action annotations for Assembly101 based on the positioning of the toy parts. Finally, we demonstrate the superior performance of our belief inference algorithm in detecting ordering mistakes on the Assembly101 dataset.

Motivation & Objective

• To develop an AI assistant capable of detecting ordering mistakes in procedural assembly tasks such as furniture or toy assembly.
• To model assembly procedures not just as sequences of actions, but as structured knowledge with spatial and temporal constraints.
• To enable real-time mistake detection by constructing belief sets incrementally as actions unfold.
• To improve upon existing anomaly or unintended action detection by incorporating context-dependent correctness.
• To provide a new coarse-level part-to-part annotation set for Assembly101 to clarify structural relationships in actions.

Proposed method

• Constructs a knowledge base with two belief sets: spatial beliefs for topological relationships between parts (e.g., wheel attaches to chassis), and temporal beliefs for prerequisite action ordering.
• Uses an online, episodic memory design (BeliefBuilder) to incrementally update beliefs as each action is observed in sequence.
• Applies a RuleBuilder to infer transitive and intransitive temporal constraints from observed mistakes and correct sequences.
• Employs a dynamic inference engine (Inferencer) that consults the belief base at each step to classify actions as correct, mistaken, corrected, or unnecessary.
• Leverages a new coarse-level annotation scheme for Assembly101 that explicitly identifies interacting part pairs in each action.
• Trains and evaluates using a 4-fold cross-validation protocol on the Assembly101 dataset with per-class precision, recall, and F1 as metrics.

Experimental results

Research questions

• RQ1 Can a knowledge base of spatial and temporal beliefs be effectively learned from real-world assembly sequences to detect ordering mistakes?
• RQ2 How does online belief construction improve mistake detection performance compared to static or batch learning methods?
• RQ3 To what extent can learned beliefs generalize across diverse action sequences and capture context-dependent correctness?
• RQ4 How does the proposed method compare to sequence-based models like LSTM and TempAgg in detecting fine-grained ordering mistakes?
• RQ5 Can the system distinguish between correction actions and correct actions, especially in cases of high class imbalance?

Key findings

• The proposed method achieves 70.6% recall and 71.8% F1 on fine-grained mistake detection, significantly outperforming LSTM (35.2% recall, 61.3% F1) and TempAgg (36.6% recall, 59.9% F1).
• The system achieves 93.1% precision on the 'correct' class, indicating high confidence in identifying valid actions.
• Unlike LSTM, which misclassifies 100% of fine-grained ordering mistakes (B, C, D) as correct, the proposed method correctly identifies these errors.
• The belief inference system generates 48 temporal rules, 23 of which are transitive, all aligned with the geometric constraints of the toy parts.
• The method reduces confusion between detach actions (E, F) and correct attach actions (A), avoiding the misclassification common in LSTM.
• The new part-to-part annotation set enables clearer modeling of structural relationships, improving the interpretability and accuracy of belief learning.