[Paper Review] UCF-101: A dataset of 101 human actions classes from videos in the wild

Khurram Soomro, Amir Zamir+1 more|arXiv (Cornell University)|Dec 3, 2012

Human Pose and Action Recognition9 references4,437 citations

TL;DR

UCF101 introduces a large, unconstrained action recognition dataset with 101 classes and 13,320 clips from YouTube, and provides baseline results using a bag-of-words approach.

ABSTRACT

We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Motivation & Objective

• Address the lack of large, diverse action datasets with realistic backgrounds and camera motion.
• Provide a large benchmark dataset (101 classes, 13k+ clips) captured from web videos to evaluate action recognition methods.
• Offer baseline recognition results to establish a reference performance on this dataset.
• Characterize the dataset's properties (group structure, clip duration, resolution, audio availability) for reproducibility and benchmarking.

Proposed method

• Construct a dataset of 101 action classes from web videos with 25 groups per action and 4–7 clips per group.
• Use 25 FPS, 320x240 resolution, DivX-encoded AVI clips with audio preserved for 51 actions.
• Extract Harris3D corners and compute 162-dim HOG/HOF descriptors.
• Build a 4000-word codebook by k-means on 100,000 STIP features and represent each clip as a 4000-dim histogram.
• Train a nonlinear multiclass SVM with histogram intersection kernel using leave-one-group-out 25-fold cross-validation.
• Report baseline accuracy and per-type confusion with a BoW baseline.

Experimental results

Research questions

• RQ1 What is the scale and diversity required for a robust action recognition benchmark in unconstrained real-world videos?
• RQ2 How does a standard bag-of-words representation perform on a large, diverse action dataset with camera motion and cluttered backgrounds?
• RQ3 What are the baseline recognition rates across different action type categories on UCF101?
• RQ4 How do clip characteristics (grouping, length, background, motion) affect recognition performance?

Key findings

• UCF101 contains 101 action classes and 13,320 clips totaling about 1600 minutes of video.
• Baseline BoW approach with 4000-word codebook achieves 44.5% overall accuracy.
• Sports actions achieve the highest baseline accuracy (~50.54%) due to distinctive motions and less cluttered backgrounds.
• Human-Object Interaction and Body-Motion Only categories have notably lower baseline accuracies (~37–38%).
• The dataset is larger and more challenging than prior datasets, with unconstrained web videos and variable clip lengths.
• The authors recommend a 25-fold cross-validation setup for consistent evaluation on UCF101.