[Paper Review] Self-Paced Video Data Augmentation with Dynamic Images Generated by Generative Adversarial Networks

Yumeng Zhang, Gaoguo Jia+3 more|arXiv (Cornell University)|Sep 16, 2019

Generative Adversarial Networks and Image Synthesis29 references4 citations

TL;DR

This paper proposes a self-paced video data augmentation method using dynamic images generated by GANs to improve video classification under data insufficiency. By compressing motion into still images, GANs generate high-quality, category-specific dynamic images, and a Self-Paced Selection (SPS) strategy filters only the most reliable samples, boosting accuracy by up to 7.2% on minority classes in HMDB4* and UCF4*.

ABSTRACT

There is an urgent need for an effective video classification method by means of a small number of samples. The deficiency of samples could be effectively alleviated by generating samples through Generative Adversarial Networks (GAN), but the generation of videos on a typical category remains to be underexplored since the complex actions and the changeable viewpoints are difficult to simulate. In this paper, we propose a generative data augmentation method for temporal stream of the Temporal Segment Networks with the dynamic image. The dynamic image compresses the motion information of video into a still image, removing the interference factors such as the background. Thus it is easier to generate images with categorical motion information using GAN. We use the generated dynamic images to enhance the features, with regularization achieved as well, thereby to achieve the effect of video augmentation. In order to deal with the uneven quality of generated images, we propose a Self-Paced Selection (SPS) method, which automatically selects the high-quality generated samples to be added to the network training. Our method is verified on two benchmark datasets, HMDB51 and UCF101. The experimental results show that the method can improve the accuracy of video classification under the circumstance of sample insufficiency and sample imbalance.

Motivation & Objective

• Address the challenge of video classification under data insufficiency and class imbalance, especially for rare or minority action categories.
• Overcome the limitations of existing GAN-based video generation, which struggles with complex actions and variable viewpoints due to high memory and computational demands.
• Develop a scalable, modular data augmentation framework that enhances temporal stream networks without requiring full video generation.
• Mitigate overfitting in few-shot learning by using GAN-generated dynamic images as regularizers with improved feature learning.
• Enable effective training on imbalanced datasets by generating synthetic samples that preserve key motion semantics while filtering out low-quality, noisy outputs.

Proposed method

• Generate dynamic images by compressing video motion into single still images using rank pooling, removing static backgrounds and preserving motion cues.
• Train a GAN on a large set of dynamic images extracted via sparse sampling from video segments, enabling efficient generation of motion-semantic images.
• Apply a Self-Paced Selection (SPS) strategy that dynamically identifies and selects only high-quality generated samples based on their category discriminability.
• Integrate the selected synthetic dynamic images into the training set at multiple training epochs, gradually increasing the number of augmented samples.
• Use the temporal stream of Temporal Segment Networks (TSN) as the backbone, leveraging the generated dynamic images to improve feature learning and decision boundary generalization.
• Apply SPS with a schedule over epochs, selecting samples at intervals to avoid early training interference and ensure progressive model improvement.

Experimental results

Research questions

• RQ1 Can GAN-generated dynamic images effectively augment video classification models when training data is scarce?
• RQ2 How can the quality variance in GAN-generated samples be mitigated to prevent negative impact on model training?
• RQ3 Does using dynamic images instead of raw video frames improve the feasibility and performance of data augmentation in video classification?
• RQ4 To what extent can synthetic dynamic images improve classification accuracy on imbalanced datasets with few samples per class?
• RQ5 Can a self-paced selection mechanism effectively identify and prioritize the most useful synthetic samples for training?

Key findings

• The proposed method improved classification accuracy by 7.2% on the UCF4* subset and 6.8% on the HMDB4* subset compared to baseline models trained on imbalanced data.
• The method achieved 91.3% accuracy on UCF101* and 60.4% on HMDB51*, outperforming state-of-the-art methods like DI Net and Two-Stream I3D when trained on imbalanced data.
• The Self-Paced Selection (SPS) mechanism successfully filtered low-quality GAN samples, improving model generalization and reducing overfitting.
• Using dynamic images instead of raw video enabled effective GAN-based augmentation, as the compressed representation preserved key motion features while minimizing background noise.
• The method significantly outperformed 3D-CNN baselines (C3D, R3D, I3D) on few-shot settings, with TSN-based models achieving 99.35% accuracy on UCF4* when augmented with SPS-GAN.
• The approach demonstrated strong generalization and modularity, showing compatibility with existing video classification frameworks like TSN without architectural changes.