[Paper Review] Do Less and Achieve More: Training CNNs for Action Recognition Utilizing Action Images from the Web

Shugao Ma, Sarah Adel Bargal+3 more|arXiv (Cornell University)|Dec 22, 2015

Human Pose and Action Recognition26 references4 citations

TL;DR

This paper proposes leveraging web-crawled and manually filtered action images to significantly boost action recognition performance in videos using CNNs, reducing reliance on large-scale video datasets. By training with both videos and discriminative action images—especially from curated or unfiltered web sources—it achieves state-of-the-art accuracy (91.1% on UCF101) with up to 10.5% absolute improvement over video-only training, even replacing half of ActivityNet’s 16.2M video frames with only 393K images.

ABSTRACT

Recently, attempts have been made to collect millions of videos to train CNN models for action recognition in videos. However, curating such large-scale video datasets requires immense human labor, and training CNNs on millions of videos demands huge computational resources. In contrast, collecting action images from the Web is much easier and training on images requires much less computation. In addition, labeled web images tend to contain discriminative action poses, which highlight discriminative portions of a video's temporal progression. We explore the question of whether we can utilize web action images to train better CNN models for action recognition in videos. We collect 23.8K manually filtered images from the Web that depict the 101 actions in the UCF101 action video dataset. We show that by utilizing web action images along with videos in training, significant performance boosts of CNN models can be achieved. We then investigate the scalability of the process by leveraging crawled web images (unfiltered) for UCF101 and ActivityNet. We replace 16.2M video frames by 393K unfiltered images and get comparable performance.

Motivation & Objective

• To investigate whether web-captured action images can improve CNN-based video action recognition performance.
• To reduce the high human and computational costs of curating large-scale video datasets for training.
• To explore the scalability and effectiveness of using unfiltered, crawled web images as a substitute for video frames.
• To evaluate whether web images provide complementary information to video data, enhancing model generalization.
• To build and release the largest action image dataset in one-to-one correspondence with UCF101 and ActivityNet for public research use.

Proposed method

• The authors collect 23.8K manually filtered web images depicting 101 UCF101 action classes, ensuring high-quality, discriminative poses.
• They train deep spatial CNNs (e.g., VGG16, VGG19) on a combination of video frames and web action images to improve feature learning.
• For scalability, they crawl 393K unfiltered web images for UCF101 and ActivityNet, using them to replace a portion of training video frames.
• They compare performance using filtered vs. unfiltered images and evaluate accuracy gains across different model depths and data ratios.
• They use standard evaluation protocols on UCF101 and ActivityNet, including mAP and top-1 accuracy, with and without optical flow features.
• They perform ablation studies to isolate the contribution of web images from model depth and motion features.

Experimental results

Research questions

• RQ1 Can web-captured action images significantly improve the performance of CNNs trained on video action recognition?
• RQ2 Does using web images reduce the need for large-scale video datasets without sacrificing accuracy?
• RQ3 How do filtered versus unfiltered web images compare in terms of performance and scalability?
• RQ4 What is the marginal gain in performance as the number of web images increases?
• RQ5 Can web images provide complementary information to video data, especially when combined with motion features?

Key findings

• Training a spatial CNN on both videos and 23.8K filtered web images achieves 83.5% accuracy on UCF101, a 10.5% absolute improvement over video-only training.
• When combined with motion features (IDT-FV), the model reaches 91.1% accuracy on UCF101, the highest reported result to date.
• Replacing half of ActivityNet’s 16.2M video frames with 393K unfiltered web images achieves comparable performance (46.3% mAP vs. 47.7% for full video set).
• Performance gains from web images diminish with increasing image count, suggesting saturation beyond a certain threshold.
• The approach is robust across different CNN architectures and remains effective even with very large video datasets like ActivityNet.
• The use of web images provides performance gains comparable to adding 9+ layers to the CNN or incorporating complex motion modeling, without increasing inference cost.