[Paper Review] Implicit Temporal Modeling with Learnable Alignment for Video Recognition

Shuyuan Tu, Qi Dai+4 more|arXiv (Cornell University)|Apr 20, 2023

Cancer-related molecular mechanisms researchBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes Implicit Learnable Alignment (ILA), a lightweight temporal modeling method for video recognition that replaces costly temporal self-attention with implicit, coarse feature alignment via learnable masks. By predicting interactive points between frame pairs and enhancing features around them, ILA implicitly models temporal dependencies, achieving 88.7% top-1 accuracy on Kinetics-400 with significantly fewer FLOPs than Swin-L and ViViT-H.

ABSTRACT

Contrastive language-image pretraining (CLIP) has demonstrated remarkable success in various image tasks. However, how to extend CLIP with effective temporal modeling is still an open and crucial problem. Existing factorized or joint spatial-temporal modeling trades off between the efficiency and performance. While modeling temporal information within straight through tube is widely adopted in literature, we find that simple frame alignment already provides enough essence without temporal attention. To this end, in this paper, we proposed a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance. Specifically, for a frame pair, an interactive point is predicted in each frame, serving as a mutual information rich region. By enhancing the features around the interactive point, two frames are implicitly aligned. The aligned features are then pooled into a single token, which is leveraged in the subsequent spatial self-attention. Our method allows eliminating the costly or insufficient temporal self-attention in video. Extensive experiments on benchmarks demonstrate the superiority and generality of our module. Particularly, the proposed ILA achieves a top-1 accuracy of 88.7% on Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H. Code is released at https://github.com/Francis-Rings/ILA .

Motivation & Objective

• To address the challenge of extending CLIP-based video recognition models with efficient and effective temporal modeling.
• To reduce reliance on complex or insufficient temporal self-attention mechanisms that increase computational cost without proportional gains.
• To explore whether implicit, coarse frame alignment can capture essential motion and action cues without explicit attention.
• To develop a plug-in module compatible with vision transformers that enhances temporal modeling with minimal computational overhead.
• To demonstrate that mutual information-rich regions between frames can serve as strong temporal signals for video recognition.

Proposed method

• Predicts an interactive point in each frame of a pair using a convolutional module conditioned on the frame pair.
• Generates a learnable mask that assigns higher weights to regions around the interactive point and lower weights elsewhere to emphasize mutual information.
• Applies the mask to weight and align features from two adjacent frames, producing coarsely aligned representations.
• Pools the aligned features into a single mutual information token for subsequent spatial self-attention.
• Integrates the ILA module into each spatial block of a vision transformer to form an Implicit Spatio-Temporal (IST) block.
• Uses the mutual information token in concatenation with other frame tokens to implicitly model temporal relations without explicit temporal attention.

Experimental results

Research questions

• RQ1 Can implicit, coarse frame alignment replace expensive temporal self-attention in CLIP-based video models without performance degradation?
• RQ2 Does identifying mutual information-rich regions between adjacent frames encode sufficient temporal signals for action recognition?
• RQ3 How does the proposed ILA method compare in efficiency and accuracy to existing temporal modeling approaches like frame-level or joint attention?
• RQ4 To what extent does the performance of ILA depend on CLIP pretraining versus the design of the alignment mechanism itself?
• RQ5 Can ILA generalize across benchmarks and achieve strong zero-shot performance on downstream video understanding tasks?

Key findings

• ILA achieves 88.7% top-1 accuracy on Kinetics-400 with only 40G FLOPs, outperforming Swin-L (87.3%) and ViViT-H (86.7%) under lower computational cost.
• ILA achieves 43.9% top-1 accuracy on SSv2 in zero-shot evaluation, surpassing X-CLIP (38.1%) and EVL (35.2%) by 5.8% and 8.7% respectively.
• On the temporal understanding benchmark without static biases, ILA achieves 81.9% top-1 (T) and a 6.1% temporal score (TS), outperforming X-CLIP (2.3 TS) and EVL (2.4 TS).
• Ablation studies show that the mutual information token via pooling and concatenation yields 81.3% accuracy, outperforming element-wise addition (80.2%) and direct concatenation (80.6%).
• ILA maintains a 0.6% top-1 accuracy advantage over Swin-B/32f even when initialized with ImageNet-21K weights, demonstrating its intrinsic superiority beyond CLIP pretraining.
• Visualization confirms that the interactive points track moving objects over time, indicating effective capture of dynamic motion cues.