[Paper Review] A Short Note about Kinetics-600

João Carreira, Eric Noland+3 more|arXiv (Cornell University)|Aug 3, 2018

Human Pose and Action Recognition5 references303 citations

TL;DR

This paper describes the Kinetics-600 dataset, an extension of Kinetics-400 to 600 classes with at least 600 clips each, and provides dataset statistics and baseline I3D results.

ABSTRACT

We describe an extension of the DeepMind Kinetics human action dataset from 400 classes, each with at least 400 video clips, to 600 classes, each with at least 600 video clips. In order to scale up the dataset we changed the data collection process so it uses multiple queries per class, with some of them in a language other than english -- portuguese. This paper details the changes between the two versions of the dataset and includes a comprehensive set of statistics of the new version as well as baseline results using the I3D neural network architecture. The paper is a companion to the release of the ground truth labels for the public test set.

Motivation & Objective

• Extend the DeepMind Kinetics dataset from 400 to 600 classes, each with at least 600 video clips.
• Improve data collection by using multiple language queries and robust similarity matching.
• Provide detailed statistics and baseline model results to aid benchmarking and research.
• Release ground truth labels for the public test set to facilitate independent evaluation.

Proposed method

• Use multiple queries per class in English and Portuguese to collect candidate videos.
• Apply weighted n-gram representations of metadata and related video titles to match videos to classes across languages.
• Maintain same pipeline as Kinetics-400: class sourcing, candidate video matching, clip sampling, human verification, and quality filtering.
• Compare Kinetics-600 to Kinetics-400 in terms of class changes and dataset size.

Experimental results

Research questions

• RQ1 How does expanding from 400 to 600 classes affect action recognition benchmark performance?
• RQ2 What data collection innovations (multilingual querying and weighted n-gram matching) improve candidate video collection for large-scale action datasets?
• RQ3 What are the dataset statistics and baseline results for Kinetics-600 compared with Kinetics-400?

Key findings

• Kinetics-600 contains 600 classes with around 500k total clips, a 50% class increase and 60% more clips than Kinetics-400.
• A baseline I3D model trained from scratch on Kinetics-600 achieves Top-1 71.7% and Top-5 90.4% on the test set.
• The Hold-out test set is harder than the standard test set, with Top-1 69.7% and Top-5 89.1%.
• Kinetics-600 includes a standard test set and a held-out test set for the Activity-Net challenge.
• About 368 of the original 400 Kinetics-400 classes are retained in Kinetics-600, with 32 renamed/modified classes and some reorganization.