[Paper Review] Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?

Hirokatsu Kataoka, Tenga Wakamiya+2 more|arXiv (Cornell University)|Apr 10, 2020

Human Pose and Action RecognitionComputer Science31 references82 citations

TL;DR

The paper investigates how mega-scale, carefully annotated video datasets (pre-training) and dataset merging affect the transfer learning performance of spatiotemporal 3D CNNs, showing gains on standard benchmarks and depth limits for very deep models.

ABSTRACT

How can we collect and use a video dataset to further improve spatiotemporal 3D Convolutional Neural Networks (3D CNNs)? In order to positively answer this open question in video recognition, we have conducted an exploration study using a couple of large-scale video datasets and 3D CNNs. In the early era of deep neural networks, 2D CNNs have been better than 3D CNNs in the context of video recognition. Recent studies revealed that 3D CNNs can outperform 2D CNNs trained on a large-scale video dataset. However, we heavily rely on architecture exploration instead of dataset consideration. Therefore, in the present paper, we conduct exploration study in order to improve spatiotemporal 3D CNNs as follows: (i) Recently proposed large-scale video datasets help improve spatiotemporal 3D CNNs in terms of video classification accuracy. We reveal that a carefully annotated dataset (e.g., Kinetics-700) effectively pre-trains a video representation for a video classification task. (ii) We confirm the relationships between #category/#instance and video classification accuracy. The results show that #category should initially be fixed, and then #instance is increased on a video dataset in case of dataset construction. (iii) In order to practically extend a video dataset, we simply concatenate publicly available datasets, such as Kinetics-700 and Moments in Time (MiT) datasets. Compared with Kinetics-700 pre-training, we further enhance spatiotemporal 3D CNNs with the merged dataset, e.g., +0.9, +3.4, and +1.1 on UCF-101, HMDB-51, and ActivityNet datasets, respectively, in terms of fine-tuning. (iv) In terms of recognition architecture, the Kinetics-700 and merged dataset pre-trained models increase the recognition performance to 200 layers with the Residual Network (ResNet), while the Kinetics-400 pre-trained model cannot successfully optimize the 200-layer architecture.

Motivation & Objective

• Assess which large-scale pre-training datasets best transfer to fine-tuning on standard video benchmarks.
• Examine how the number of categories and instances in pre-training affects performance.
• Test simple dataset merging to increase pre-training data and evaluate its impact.
• Explore the effect of increasing model depth (number of layers) for 3D CNNs under different pre-training regimes.
• Compare 3D-ResNet and (2+1)D architectures under mega-scale pre-training.

Proposed method

• Pre-train 3D-ResNet variants on Kinetics-700, MiT, STAIR, and Mini-HVU datasets.
• Fine-tune on UCF-101, HMDB-51, and ActivityNet to measure transfer performance.
• Systematically vary #category and #instance to study data amount effects on accuracy.
• Create merged pre-training datasets (e.g., K+M, K+M+S) and compare to single-dataset pre-training.
• Evaluate model depth (ResNet-18 to ResNet-200) and compare 3D-ResNet with (2+1)D variants.
• Compare results with and without optical-flow streams (note: this work focuses on 3D CNNs with single-stream inputs).

Experimental results

Research questions

• RQ1 What pre-training datasets best transfer to standard video recognition benchmarks for 3D CNNs?
• RQ2 How do the numbers of categories vs. instances in pre-training influence transfer accuracy?
• RQ3 Does simply merging public video datasets to form larger pre-training sets improve fine-tuning performance?
• RQ4 How does increasing model depth affect transfer performance under different pre-training regimes?

Key findings

• Kinetics-700 pre-training yields the best transfer performance among single-dataset pre-training options on UCF-101, HMDB-51, and ActivityNet (top-1 video-level accuracy).
• Merging Kinetics-700 with MiT (K+M) further improves fine-tuning results, e.g., gains of +0.9 (UCF-101), +3.4 (HMDB-51), and +1.1 (ActivityNet) over Kinetics-700 baseline.
• A deeper 3D-ResNet (e.g., ResNet-200) benefits from Kinetics-700 and K+M pre-training, achieving higher accuracy on UCF-101, HMDB-51, and ActivityNet, while Kinetics-400 pre-training does not reliably improve very deep models.
• RGB-only 3D CNNs (and their 2+1D counterparts) show stronger transfer when pre-trained on larger, well-annotated datasets; simply increasing data size does not always help (domain mismatch can occur).
• Tabled results show specific gains from pre-training choices, e.g., R3D-50 with Kinetics-700: UCF-101 92.0, HMDB-51 66.0, ActivityNet 75.9; with K+M: 92.9, 69.4, 77.0; R(2+1)D-50 with Kinetics-700: 93.4, 69.4, 78.4.
• The Kinetics-700 dataset generally outperforms other single datasets like MiT or STAIR for transfer learning in this setup.