[Paper Review] The Kinetics Human Action Video Dataset

Will Kay, João Carreira+10 more|arXiv (Cornell University)|May 19, 2017

Human Pose and Action Recognition4 references2,882 citations

TL;DR

Describes the DeepMind Kinetics dataset with 400 action classes and ~306k video clips from YouTube for human action classification, plus baseline results for several architectures and a preliminary bias analysis.

ABSTRACT

We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.

Motivation & Objective

• Provide a large-scale, diverse benchmark for human action classification to advance deep learning methods.
• Ensure clips are sourced from diverse YouTube videos to maximize variation in actors, viewpoints, lighting, and backgrounds.
• Offer baseline model performances to guide architectural choices for video understanding.
• Analyze potential dataset biases (e.g., gender or demographic biases) and their impact on classifiers.

Proposed method

• Curate 400 action classes spanning singular, pairwise (person–person), and person–object actions from YouTube.
• Collect candidate clips via YouTube searches and temporal localization using image classifiers trained from image search feedback.
• Employ Mechanical Turk for human verification to ensure clips depict the target action, with multiple positive confirmations required.
• Deduplicate clips across videos via Inception-V1 feature similarities and cosine thresholding to ensure one clip per video.
• Train and evaluate three baseline video architectures (ConvNet+LSTM, Two-Stream, and 3D-ConvNet) from scratch on Kinetics, with ImageNet pre-training where applicable, and report top-1/top-5 results.
• Discuss dataset biases and perform preliminary analyses on gender balance and potential classifier bias.

Experimental results

Research questions

• RQ1 How large and diverse can a video action dataset be to robustly train deep networks for action classification?
• RQ2 What baseline performance do standard video architectures achieve on Kinetics, and how does this compare to existing datasets like UCF-101 and HMDB-51?
• RQ3 Does the Kinetics dataset exhibit gender or demographic biases that affect classifier fairness, and how detectable are such biases with preliminary analysis?
• RQ4 What data collection and cleaning strategies are effective for assembling a large-scale, high-quality action dataset from web videos?

Key findings

• Kinetics contains 400 action classes with 400–1150 clips per class, totaling 306,245 videos and 306,245 clips, with each clip ~10 seconds.
• Baseline architectures trained from scratch on Kinetics show lower performance than on UCF-101 and HMDB-51, highlighting the dataset’s greater difficulty and diversity.
• Three baseline models were evaluated: ConvNet+LSTM, Two-Stream, and a 3D ConvNet, with varying input modalities and temporal footprints, demonstrating the trade-offs between temporal modeling and computational cost.
• Preliminary bias analysis indicates no strong evidence of gender-based classifier bias across most classes, though some exceptions exist (e.g., “shaving beard” male-dominated, “dunking basketball” male-dominated, and “filling eyebrows” female-dominated) and warrant deeper study.
• The paper provides baseline TensorFlow implementations and discusses preprocessing, data augmentation, and training regimens to enable reproducibility.