[Paper Review] Towards Good Practices for Very Deep Two-Stream ConvNets

Limin Wang, Yuanjun Xiong+2 more|arXiv (Cornell University)|Jul 8, 2015

Human Pose and Action Recognition19 references385 citations

TL;DR

This paper proposes very deep two-stream ConvNets for video action recognition by adapting deep ImageNet architectures (GoogLeNet, VGGNet) to the video domain, employing specialized training practices to combat overfitting on small datasets. It achieves a new state-of-the-art accuracy of 91.4% on UCF101 using pre-training, data augmentation, low learning rates, and high dropout.

ABSTRACT

Deep convolutional networks have achieved great success for object recognition in still images. However, for action recognition in videos, the improvement of deep convolutional networks is not so evident. We argue that there are two reasons that could probably explain this result. First the current network architectures (e.g. Two-stream ConvNets) are relatively shallow compared with those very deep models in image domain (e.g. VGGNet, GoogLeNet), and therefore their modeling capacity is constrained by their depth. Second, probably more importantly, the training dataset of action recognition is extremely small compared with the ImageNet dataset, and thus it will be easy to over-fit on the training dataset. To address these issues, this report presents very deep two-stream ConvNets for action recognition, by adapting recent very deep architectures into video domain. However, this extension is not easy as the size of action recognition is quite small. We design several good practices for the training of very deep two-stream ConvNets, namely (i) pre-training for both spatial and temporal nets, (ii) smaller learning rates, (iii) more data augmentation techniques, (iv) high drop out ratio. Meanwhile, we extend the Caffe toolbox into Multi-GPU implementation with high computational efficiency and low memory consumption. We verify the performance of very deep two-stream ConvNets on the dataset of UCF101 and it achieves the recognition accuracy of $91.4\%$.

Motivation & Objective

• Address the limited performance of deep ConvNets in video action recognition due to shallow architectures and small training datasets.
• Overcome overfitting in small video datasets by designing effective training practices for very deep two-stream networks.
• Extend the Caffe toolbox to multi-GPU training with high efficiency and low memory consumption for scalable deep learning in video tasks.
• Demonstrate state-of-the-art performance on UCF101 by combining very deep network architectures with robust training strategies.

Proposed method

• Adapt very deep ImageNet architectures (GoogLeNet and VGGNet) to the video domain by applying them to spatial and temporal streams.
• Implement pre-training on ImageNet for both spatial and temporal networks to improve initialization and generalization.
• Use smaller learning rates and higher dropout ratios to reduce overfitting during training on small video datasets.
• Apply extensive data augmentation techniques to increase effective training data diversity and robustness.
• Implement a multi-GPU version of Caffe with high computational efficiency and low memory consumption for scalable training.
• Fusion strategy combines spatial and temporal network predictions using a weighted linear combination (2:1 for temporal to spatial).

Experimental results

Research questions

• RQ1 Can very deep two-stream ConvNets achieve superior performance in action recognition when adapted from image classification models?
• RQ2 What specific training practices are necessary to prevent overfitting when training very deep networks on small video datasets like UCF101?
• RQ3 How do architectural depth and training methodology jointly affect recognition accuracy in video action recognition?
• RQ4 To what extent do pre-training, data augmentation, and regularization improve performance on limited video datasets?
• RQ5 Can the Caffe deep learning framework be effectively extended to support efficient multi-GPU training for very deep two-stream networks?

Key findings

• The proposed very deep two-stream ConvNets achieve a state-of-the-art recognition accuracy of 91.4% on the UCF101 dataset.
• VGGNet-16 outperforms shallower architectures (e.g., ClarifaiNet, GoogLeNet) by approximately 5% on spatial streams and 4% on temporal streams.
• Very deep two-stream networks improve over the original two-stream ConvNets by 3.4% accuracy, demonstrating the benefit of increased depth.
• The performance gap between the proposed method and prior methods (e.g., TDD+FV at 90.3%) is 1.1%, confirming its superiority.
• Without the proposed good practices, deeper models on THUMOS15 failed to generalize, indicating that the training strategies are essential for success.
• The multi-GPU Caffe implementation enables efficient training with low memory consumption, supporting large-scale deep learning in video tasks.