[Paper Review] Human Action Recognition in Drone Videos using a Few Aerial Training Examples

Waqas Sultani, Mubarak Shah|arXiv (Cornell University)|Oct 22, 2019

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a disjoint multitask learning framework that leverages synthetic data from video games and GAN-generated aerial features to improve human action recognition in drone videos when only a few real aerial training examples are available. By integrating game videos and GAN-generated features through alternating training, the method achieves significant accuracy gains—up to 68.2% average accuracy on YouTube-Aerial and 35.9% on UCF-ARG—demonstrating that synthetic data can effectively compensate for real data scarcity in aerial action recognition.

ABSTRACT

Drones are enabling new forms of human actions surveillance due to their low cost and fast mobility. However, using deep neural networks for automatic aerial action recognition is difficult due to the need for a large number of training aerial human action videos. Collecting a large number of human action aerial videos is costly, time-consuming, and difficult. In this paper, we explore two alternative data sources to improve aerial action classification when only a few training aerial examples are available. As a first data source, we resort to video games. We collect plenty of aerial game action videos using two gaming engines. For the second data source, we leverage conditional Wasserstein Generative Adversarial Networks to generate aerial features from ground videos. Given that both data sources have some limitations, e.g. game videos are biased towards specific actions categories (fighting, shooting, etc.,), and it is not easy to generate good discriminative GAN-generated features for all types of actions, we need to efficiently integrate two dataset sources with few available real aerial training videos. To address this challenge of the heterogeneous nature of the data, we propose to use a disjoint multitask learning framework. We feed the network with real and game, or real and GAN-generated data in an alternating fashion to obtain an improved action classifier. We validate the proposed approach on two aerial action datasets and demonstrate that features from aerial game videos and those generated from GAN can be extremely useful for an improved action recognition in real aerial videos when only a few real aerial training examples are available.

Motivation & Objective

• To address the challenge of limited real aerial training videos for human action recognition in drone footage.
• To explore alternative data sources—video games and GAN-generated features—to augment scarce real aerial data.
• To develop a robust method that integrates heterogeneous data sources (game videos and GAN features) with real aerial examples.
• To improve generalization and accuracy in aerial action classification despite domain shift and label discrepancies between data sources.
• To validate the effectiveness of the proposed framework on benchmark aerial action recognition datasets.

Proposed method

• Collect large-scale aerial game videos using photo-realistic gaming engines like GTA and FIFA to simulate real-world drone environments.
• Use conditional Wasserstein GANs to generate discriminative aerial-style features from ground-view videos, simulating drone camera perspectives.
• Design a disjoint multitask learning framework that alternately trains on real, game, and GAN-generated data to handle label and domain discrepancies.
• Train the action classifier using a combination of real aerial videos, game videos, and GAN-generated features, with no requirement for aligned action labels across sources.
• Employ multiple 3D CNN backbones (MFN-3D, I3D, ResNet-3D) to evaluate the robustness and generalization of the proposed method across architectures.
• Use confusion matrices and class-wise accuracy analysis to evaluate the reduction in misclassification between similar actions.

Experimental results

Research questions

• RQ1 Can video game videos serve as a viable synthetic data source for improving action recognition in real drone videos?
• RQ2 Can GAN-generated aerial features effectively augment real training data when only a few real examples are available?
• RQ3 How can heterogeneous data sources (game videos and GAN-generated features) with different action labels be effectively integrated into a single action recognition model?
• RQ4 Does disjoint multitask learning improve classification accuracy and reduce confusion between similar actions in aerial video action recognition?
• RQ5 To what extent does the proposed method generalize across different 3D CNN architectures and benchmark datasets?

Key findings

• The proposed method achieved 68.2% average accuracy on the YouTube-Aerial dataset when training with both game videos and GAN-generated features via disjoint multitask learning, a significant improvement over training with real data alone (51.3%) or with only one synthetic source.
• On the UCF-ARG dataset, the method achieved 35.92% average accuracy using both game and GAN data, outperforming training with ground videos (21.33%) or either synthetic source alone.
• For actions with strong motion patterns such as Biking, Swing, Kayaking, Carry, Clap, and Running, the method achieved the highest gains, with accuracy increases of up to 60% on specific classes.
• Confusion matrices showed a substantial reduction in misclassification between similar actions, with overall accuracy rising from 51.3% (no DML) to 64.5% (with DML using both data sources).
• The method outperformed baselines in 5 out of 8 classes on YouTube-Aerial and 6 out of 10 classes on UCF-ARG when using both synthetic data sources, indicating strong generalization.
• The approach was particularly effective for actions with clear motion cues but less effective for actions with strong background biases (e.g., Diving) or minimal body motion (e.g., Skateboarding).