[Paper Review] MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition

Ruiqi Xian, Xijun Wang+1 more|arXiv (Cornell University)|Mar 5, 2023

Human Pose and Action Recognition4 citations

TL;DR

MITFAS proposes a mutual information-based temporal feature alignment and frame sampling method for aerial video action recognition, addressing challenges from small human scale, viewpoint changes, and occlusions in UAV videos. By aligning motion-relevant features across frames and selecting the most informative frame sequences using joint mutual information, MITFAS achieves 18.9% higher Top-1 accuracy than SOTA on UAV-Human, 7.3% on Drone-Action, and 7.16% on NEC Drones.

ABSTRACT

We present a novel approach for action recognition in UAV videos. Our formulation is designed to handle occlusion and viewpoint changes caused by the movement of a UAV. We use the concept of mutual information to compute and align the regions corresponding to human action or motion in the temporal domain. This enables our recognition model to learn from the key features associated with the motion. We also propose a novel frame sampling method that uses joint mutual information to acquire the most informative frame sequence in UAV videos. We have integrated our approach with X3D and evaluated the performance on multiple datasets. In practice, we achieve 18.9% improvement in Top-1 accuracy over current state-of-the-art methods on UAV-Human(Li et al., 2021), 7.3% improvement on Drone-Action(Perera et al., 2019), and 7.16% improvement on NEC Drones(Choi et al., 2020).

Motivation & Objective

• Address the challenge of small, variable-scale human actors in UAV videos due to high-altitude camera views.
• Overcome difficulties caused by viewpoint changes, occlusions, and dynamic backgrounds in aerial video action recognition.
• Improve model focus on human motion features rather than background changes during training.
• Develop a frame sampling strategy that selects the most informative and distinctive frames for training.
• Enable effective integration with existing video recognition backbones like X3D without requiring retraining or domain-specific adaptation.

Proposed method

• Use mutual information as a criterion to align spatial-temporal features between consecutive frames, maximizing similarity in motion-relevant regions.
• Apply a pixel-level feature alignment technique that estimates optimal transformation parameters (ω*) to align human action regions across frames.
• Introduce a reference frame-based iterative alignment process, where each subsequent frame is aligned to the previous aligned region.
• Propose a novel frame sampling method based on joint mutual information to select the most informative and diverse frame sequences.
• Formulate the frame selection criterion using a weighted combination of mutual information and joint mutual information (Eq. 11), with hyperparameters α and β.
• Integrate the proposed alignment and sampling modules with the X3D backbone for end-to-end video action recognition.

Figure 1 : $F_{t}$ and $F_{t+1}$ are two frames at time $t$ and $t+1$ , respectively, from the same UAV video. The human actor in the two frames occupies less than 10% of the pixels due to the high camera altitude (top images). (a) MITFAS will focus on the regions corresponding to salient motions an

Experimental results

Research questions

• RQ1 How can mutual information be effectively leveraged to align motion-relevant features across frames in UAV videos with high camera altitude and viewpoint variation?
• RQ2 What frame sampling strategy based on mutual information yields the most informative and diverse frame sequences for training aerial action recognition models?
• RQ3 To what extent does temporal feature alignment improve recognition accuracy when combined with standard video recognition backbones like X3D?
• RQ4 How does the proposed method compare to uniform, random, and adaptive sampling baselines in terms of Top-1 accuracy on UAV video benchmarks?
• RQ5 Can mutual information serve as a superior similarity measure compared to other metrics like L2 distance or cosine similarity in the context of UAV video action recognition?

Key findings

• MITFAS achieves 18.9% higher Top-1 accuracy than the current state-of-the-art method on the UAV-Human dataset, with a 20.2% improvement over the baseline X3D.
• On the Drone-Action dataset, MITFAS improves Top-1 accuracy by 7.3% over SOTA and 16.6% over the baseline X3D with uniform sampling.
• On the NEC Drones dataset, MITFAS achieves 78.62% Top-1 accuracy, representing a 7.18% improvement over the previous SOTA.
• The proposed temporal feature alignment (TFA) improves Top-1 accuracy by 16–17.5% when integrated with X3D and various sampling methods.
• Mutual information outperforms other similarity measures (e.g., L2, cosine) in frame similarity evaluation, achieving the highest accuracy on UAV-Human (Table 8).
• The optimal hyperparameters for the sampling criterion are α = 1.0 and β = 1.0, indicating that both mutual information and joint mutual information components are equally important for frame selection.

Figure 2 : Given a starting frame $F_{t}$ in a UAV video, we use a localization network to localize the human action and crop the region containing the human motion as the reference image $F_{r}$ . At time $t+1$ , we use our feature alignment algorithm to estimate the optimal operation parameter $\o