[Paper Review] Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities

Fadime Şener, Dibyadip Chatterjee+5 more|arXiv (Cornell University)|Mar 28, 2022

Human Pose and Action Recognition4 citations

TL;DR

Assembly101 introduces a large-scale, multi-view video dataset of 4321 sequences capturing people assembling and disassembling 101 take-apart toy vehicles from 8 static and 4 egocentric viewpoints. It features over 1 million fine-grained action segments, 18 million 3D hand poses, and introduces a novel mistake detection task, enabling benchmarking in action recognition, anticipation, temporal segmentation, and generalization across toys, views, and skill levels with strong performance gains via multi-view fusion.

ABSTRACT

Assembly101 is a new procedural activity dataset featuring 4321 videos of people assembling and disassembling 101 "take-apart" toy vehicles. Participants work without fixed instructions, and the sequences feature rich and natural variations in action ordering, mistakes, and corrections. Assembly101 is the first multi-view action dataset, with simultaneous static (8) and egocentric (4) recordings. Sequences are annotated with more than 100K coarse and 1M fine-grained action segments, and 18M 3D hand poses. We benchmark on three action understanding tasks: recognition, anticipation and temporal segmentation. Additionally, we propose a novel task of detecting mistakes. The unique recording format and rich set of annotations allow us to investigate generalization to new toys, cross-view transfer, long-tailed distributions, and pose vs. appearance. We envision that Assembly101 will serve as a new challenge to investigate various activity understanding problems.

Motivation & Objective

• To create a large-scale, multi-view video dataset of natural, goal-oriented procedural activities outside the kitchen domain.
• To capture rich variations in action ordering, mistakes, and corrections during real-world assembly and disassembly tasks.
• To enable benchmarking on action recognition, anticipation, temporal segmentation, and a novel mistake detection task.
• To support research in cross-view generalization, long-tailed distribution robustness, and pose vs. appearance understanding.
• To provide synchronized static and egocentric views for improved 3D understanding of hand-object interactions.

Proposed method

• The dataset was collected using 8 static cameras and 4 egocentric cameras to capture synchronized multi-view footage of participants assembling and disassembling 101 toy vehicles.
• Each sequence is annotated with 1380 fine-grained and 202 coarse action classes, over 100K coarse and 1M fine-grained action segments, and 18 million 3D hand poses.
• For action recognition and anticipation, models like MS-TCN++ and C2F-TCN were trained using frame-wise cross-entropy loss with class weighting and mean-square error loss.
• For mistake detection, a modified TempAgg model was used with 60-second context windows and multi-scale temporal aggregation to detect errors and corrections.
• Multi-view fusion was implemented via average-pooling of scores from fixed and egocentric views to improve recognition and anticipation performance.
• The training pipeline used TSM features with Adam optimization, learning rate decay, and dropout for robustness to class imbalance.

Experimental results

Research questions

• RQ1 How does multi-view supervision from synchronized static and egocentric cameras improve action recognition and anticipation performance?
• RQ2 To what extent can models generalize to unseen toy categories, especially under long-tailed distributions of action classes?
• RQ3 Can models effectively detect mistakes and corrections in unscripted, free-style procedural activities?
• RQ4 How do pose-based and appearance-based features compare in understanding hand-object interactions in procedural tasks?
• RQ5 What is the impact of skill level and action ordering variation on model performance in procedural activity understanding?

Key findings

• Multi-view fusion significantly improved action recognition Top-5 accuracy to 71.6% (Fusion) compared to 58.5% (Overall) on fine-grained actions.
• The best-performing model achieved 88.5% Top-5 accuracy on fixed views and 82.7% on egocentric views, with fusion boosting performance to 88.5%.
• Mistake detection was effectively modeled using TempAgg with 60-second context windows and multi-scale aggregation, achieving improved recall on error and correction classes.
• The 'transporter' toy category achieved the highest recognition accuracy (91.2% Top-5) due to high recording density with 22 participants.
• Verbs like 'pick up' and 'put down' had the highest recognition recall, while 'attempt to' verbs had the lowest, indicating challenges in modeling intent.
• Coarse action recognition showed that 'demonstrate', 'attach', and 'detach' had the highest recall, while 'position', 'remove', and 'attempt to screw' were the most challenging verbs.