[Paper Review] Look More but Care Less in Video Recognition

Yitian Zhang, Yue Bai+3 more|arXiv (Cornell University)|Nov 18, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes Ample and Focal Network (AFNet), a two-branch 2D-CNN architecture that enables efficient video recognition by processing all input frames with minimal computation while dynamically focusing on salient frames via a lightweight navigation module. The method achieves higher accuracy than baseline methods even with fewer frames by implicitly modeling temporal dynamics through adaptive frame selection within the network, reducing computational cost without sacrificing information.

ABSTRACT

Existing action recognition methods typically sample a few frames to represent each video to avoid the enormous computation, which often limits the recognition performance. To tackle this problem, we propose Ample and Focal Network (AFNet), which is composed of two branches to utilize more frames but with less computation. Specifically, the Ample Branch takes all input frames to obtain abundant information with condensed computation and provides the guidance for Focal Branch by the proposed Navigation Module; the Focal Branch squeezes the temporal size to only focus on the salient frames at each convolution block; in the end, the results of two branches are adaptively fused to prevent the loss of information. With this design, we can introduce more frames to the network but cost less computation. Besides, we demonstrate AFNet can utilize fewer frames while achieving higher accuracy as the dynamic selection in intermediate features enforces implicit temporal modeling. Further, we show that our method can be extended to reduce spatial redundancy with even less cost. Extensive experiments on five datasets demonstrate the effectiveness and efficiency of our method.

Motivation & Objective

• Address the trade-off between computational cost and recognition accuracy in video recognition by enabling the use of more frames without proportional computation increase.
• Overcome the limitations of existing methods that discard unimportant frames early, leading to information loss and reduced model robustness.
• Introduce a dynamic, end-to-end trainable frame selection mechanism within the network to enhance temporal modeling and feature representation.
• Design a lightweight navigation module that enables selective computation on salient frames at each convolutional block, reducing redundancy without introducing policy networks.
• Extend the framework to handle spatial redundancy by adapting the selection mechanism to spatial regions, further improving efficiency.

Proposed method

• Propose a two-branch architecture: the Ample Branch processes all input frames with downsampling and channel reduction to preserve full temporal information with low computation.
• Introduce the Focal Branch, which uses a navigation module to dynamically select and compute only on salient frames at each convolutional block, reducing temporal computation.
• Design a lightweight navigation module that generates frame selection weights based on intermediate features, enabling end-to-end training without additional policy networks.
• Implement adaptive feature fusion between the two branches to combine global context from the Ample Branch and focused representations from the Focal Branch.
• Enable extension to spatial redundancy reduction by applying the same dynamic selection mechanism to spatial regions of selected frames.
• Train the entire network end-to-end, allowing the navigation module to learn task-specific frame importance through backpropagation.

Experimental results

Research questions

• RQ1 Can a dual-branch network design preserve full input frame information while reducing computational cost in video recognition?
• RQ2 Does dynamic frame selection within the network, rather than at input, lead to better temporal modeling and improved accuracy with fewer frames?
• RQ3 Can a lightweight, learnable navigation module replace complex policy networks for frame selection while maintaining end-to-end training and efficiency?
• RQ4 To what extent can the proposed method reduce both temporal and spatial redundancy without sacrificing recognition performance?
• RQ5 How does the dynamic selection strategy at intermediate layers compare to fixed sampling or random selection in terms of accuracy and efficiency?

Key findings

• AFNet achieves 75.9% top-1 accuracy on Something-Something V1 with only 50% of frames selected, outperforming the baseline with 100% frames and 74.6% accuracy.
• The two-branch design with adaptive fusion improves accuracy by up to 3.4% over single-branch models when using the same number of frames.
• The navigation module consistently outperforms fixed sampling strategies (random, uniform, normal) across all frame selection ratios, especially at low ratios (e.g., 0.3), where the accuracy gap is most pronounced.
• The method reduces computation cost by focusing on salient frames while preserving all input information, enabling higher accuracy with fewer frames compared to static sampling.
• The dynamic selection mechanism implicitly models temporal relationships, allowing the network to learn temporal dependencies more effectively than static methods.
• The framework can be extended to reduce spatial redundancy, further improving efficiency when combined with spatial adaptive methods, as shown in ablation studies.