[Paper Review] SLAC: A Sparsely Labeled Dataset for Action Classification and Localization

Hang Zhao, Zhicheng Yan+3 more|arXiv (Cornell University)|Dec 26, 2017

Human Pose and Action Recognition22 citations

TL;DR

This paper introduces SLAC, a large-scale video dataset with 520K untrimmed videos and 1.75M sparse clip annotations across 200 action classes, created via a scalable framework that reduces human labeling time by 95% by focusing annotators on hard, ambiguous clips. Pre-training on SLAC boosts action recognition accuracy by up to 35.4% on HMDB-51 and improves action localization mAP by 8.6% on THUMOS14.

ABSTRACT

This paper describes a procedure for the creation of large-scale video datasets for action classification and localization from unconstrained, realistic web data. The scalability of the proposed procedure is demonstrated by building a novel video benchmark, named SLAC (Sparsely Labeled ACtions), consisting of over 520K untrimmed videos and 1.75M clip annotations spanning 200 action categories. Using our proposed framework, annotating a clip takes merely 8.8 seconds on average. This represents a saving in labeling time of over 95% compared to the traditional procedure of manual trimming and localization of actions. Our approach dramatically reduces the amount of human labeling by automatically identifying hard clips, i.e., clips that contain coherent actions but lead to prediction disagreement between action classifiers. A human annotator can disambiguate whether such a clip truly contains the hypothesized action in a handful of seconds, thus generating labels for highly informative samples at little cost. We show that our large-scale dataset can be used to effectively pre-train action recognition models, significantly improving final metrics on smaller-scale benchmarks after fine-tuning. On Kinetics, UCF-101 and HMDB-51, models pre-trained on SLAC outperform baselines trained from scratch, by 2.0%, 20.1% and 35.4% in top-1 accuracy, respectively when RGB input is used. Furthermore, we introduce a simple procedure that leverages the sparse labels in SLAC to pre-train action localization models. On THUMOS14 and ActivityNet-v1.3, our localization model improves the mAP of baseline model by 8.6% and 2.5%, respectively.

Motivation & Objective

• To address the high cost and scalability limitations of creating large-scale video datasets for action recognition and localization.
• To reduce human labeling effort by identifying and prioritizing only the most ambiguous, hard-to-classify clips for human annotation.
• To develop a scalable pipeline that leverages web-scale, unconstrained video data for dataset creation.
• To demonstrate that pre-training on SLAC significantly improves performance on downstream action recognition and localization benchmarks.

Proposed method

• Automatically extract candidate clips from untrimmed web videos using pre-trained action classifiers.
• Identify 'hard clips'—those causing prediction disagreement among multiple classifiers—as high-priority for human annotation.
• Use human annotators to label only these hard clips, which take on average 8.8 seconds per clip, drastically reducing labeling cost.
• Leverage the resulting sparse labels to pre-train both action recognition and action localization models.
• Apply a simple procedure to adapt sparse clip-level labels for weakly supervised action localization.
• Fine-tune pre-trained models on standard benchmarks to evaluate transfer performance.

Experimental results

Research questions

• RQ1 Can a scalable, automated pipeline reduce human labeling effort in large-scale video dataset creation while maintaining label quality?
• RQ2 How effective is pre-training on SLAC for improving action recognition performance on downstream benchmarks?
• RQ3 Can sparse labels from SLAC be effectively used to train action localization models?
• RQ4 Does focusing human annotation on hard clips yield better model performance compared to random or uniformly sampled labeling?
• RQ5 What performance gains does SLAC enable in terms of top-1 accuracy and mAP on standard benchmarks?

Key findings

• Pre-training on SLAC improves top-1 accuracy by 2.0% on Kinetics, 20.1% on UCF-101, and 35.4% on HMDB-51 when using RGB input.
• The proposed labeling framework reduces average annotation time to 8.8 seconds per clip, achieving over 95% savings compared to traditional manual trimming and localization.
• Action recognition models pre-trained on SLAC outperform those trained from scratch on all evaluated benchmarks.
• The weakly supervised localization method based on SLAC's sparse labels improves mAP by 8.6% on THUMOS14 and 2.5% on ActivityNet-v1.3.
• The framework successfully scales to over 520K untrimmed videos and 1.75M clip annotations across 200 action categories.
• The use of hard clip prioritization ensures high label informativeness with minimal human effort.