[Paper Review] PAN: Towards Fast Action Recognition via Learning Persistence of Appearance

Can Zhang, Yuexian Zou+2 more|arXiv (Cornell University)|Aug 8, 2020

Human Pose and Action Recognition49 references32 citations

TL;DR

PAN presents a fast, end-to-end framework for action recognition that replaces optical flow with a lightweight Persistence of Appearance (PA) cue and a various-timescale aggregation pooling (VAP) to model long-term dynamics, achieving real-time capable performance with strong accuracy.

ABSTRACT

Efficiently modeling dynamic motion information in videos is crucial for action recognition task. Most state-of-the-art methods heavily rely on dense optical flow as motion representation. Although combining optical flow with RGB frames as input can achieve excellent recognition performance, the optical flow extraction is very time-consuming. This undoubtably will count against real-time action recognition. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. Our motivation lies in the observation that small displacements of motion boundaries are the most critical ingredients for distinguishing actions, so we design a novel motion cue called Persistence of Appearance (PA). In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries. Also, it is more efficient by only accumulating pixel-wise differences in feature space, instead of using exhaustive patch-wise search of all the possible motion vectors. Our PA is over 1000x faster (8196fps vs. 8fps) than conventional optical flow in terms of motion modeling speed. To further aggregate the short-term dynamics in PA to long-term dynamics, we also devise a global temporal fusion strategy called Various-timescale Aggregation Pooling (VAP) that can adaptively model long-range temporal relationships across various timescales. We finally incorporate the proposed PA and VAP to form a unified framework called Persistent Appearance Network (PAN) with strong temporal modeling ability. Extensive experiments on six challenging action recognition benchmarks verify that our PAN outperforms recent state-of-the-art methods at low FLOPs. Codes and models are available at: https://github.com/zhang-can/PAN-PyTorch.

Motivation & Objective

• Motivate fast action recognition by reducing reliance on time-consuming optical flow.
• Introduce Persistence of Appearance (PA) as an efficient motion cue focusing on motion boundaries.
• Develop a lightweight PA module and integrate it with a temporal fusion strategy (VAP) to form PAN.
• Explore two encoding schemes for PA and compare their performance and efficiency.
• Demonstrate PAN's effectiveness across temporal-dominant and scene-dominant action datasets.

Proposed method

• Define PA by extending brightness constancy from image space to feature space, using pixel-wise feature differences to capture boundary motion.
• Implement a PA module with a single low-level conv layer (8x7×7 filters) to produce a PA saliency map from adjacent frames.
• Use two encoding schemes: e1 (PA as motion modality) and e2 (PA as attention); e1 directly feeds PA into the backbone, while e2 uses PA to modulate appearance features.
• Construct PAN with two variants: PAN Full (separate RGB and PA branches with later fusion) and PAN Lite (unified backbone processing concatenated RGB+PA inputs).
• Introduce Various-timescale Aggregation Pooling (VAP) to fuse long-range temporal information via multi-timescale pooling and adaptive weighting across timescales.

Experimental results

Research questions

• RQ1 Can Persistence of Appearance (PA) capture equivalent motion information more efficiently than optical flow for action recognition?
• RQ2 How does PA contribute to temporal modeling when integrated with deep backbones and a global temporal fusion strategy (VAP)?
• RQ3 Which PA encoding scheme yields better recognition performance and efficiency in PAN?
• RQ4 Do PAN Full and PAN Lite offer complementary or trade-off performance on temporal-dominant vs. scene-dominant datasets?

Key findings

• PA is highly efficient (8196 fps) and achieves strong accuracy (89.5%) on UCF101 split 1, outpacing several optical-flow-based approaches in runtime while maintaining competitive accuracy.
• Encoding PA as motion modality (e1) outperforms PA as attention (e2) in both accuracy and efficiency on UCF101; e2 is slower and slightly less accurate.
• PAN Full and PAN Lite achieve strong results across temporal-dominant and scene-d dominant datasets, validating fast action recognition without precomputed optical flow.
• VAP enables effective long-range temporal modeling across multiple timescales with lightweight parameters, improving video-level representations.
• Compared to traditional optical-flow methods (TV-L1, FlowNet variants), PA offers substantially higher speed with competitive accuracy.