[Paper Review] Temporal Query Networks for Fine-grained Video Understanding

Chuhan Zhang, Ankush Gupta+1 more|arXiv (Cornell University)|Apr 19, 2021

Human Pose and Action Recognition24 references4 citations

TL;DR

This paper proposes the Temporal Query Network (TQN), a Transformer-based architecture that enables fine-grained video action recognition by treating each action class as a query to be answered from untrimmed videos. Using temporal attention and a stochastic feature bank for dense, memory-efficient training, TQN achieves state-of-the-art performance on FineGym and Diving48 using only RGB features, demonstrating superior modeling of short, subtle actions across variable durations.

ABSTRACT

Our objective in this work is fine-grained classification of actions in untrimmed videos, where the actions may be temporally extended or may span only a few frames of the video. We cast this into a query-response mechanism, where each query addresses a particular question, and has its own response label set. We make the following four contributions: (I) We propose a new model - a Temporal Query Network - which enables the query-response functionality, and a structural understanding of fine-grained actions. It attends to relevant segments for each query with a temporal attention mechanism, and can be trained using only the labels for each query. (ii) We propose a new way - stochastic feature bank update - to train a network on videos of various lengths with the dense sampling required to respond to fine-grained queries. (iii) We compare the TQN to other architectures and text supervision methods, and analyze their pros and cons. Finally, (iv) we evaluate the method extensively on the FineGym and Diving48 benchmarks for fine-grained action classification and surpass the state-of-the-art using only RGB features.

Motivation & Objective

• To address fine-grained action recognition in untrimmed videos where actions are temporally brief and visually subtle, requiring precise temporal localization.
• To enable query-based video understanding where each action class is treated as a distinct question to be answered from the video.
• To train a model effectively on long, untrimmed videos with dense temporal sampling, despite GPU memory constraints.
• To develop a training framework that supports weakly supervised learning, where only query-level labels are available without temporal annotations.
• To surpass existing methods in accuracy on fine-grained benchmarks using only RGB features, avoiding expensive optical flow or 3D convolutions.

Proposed method

• The Temporal Query Network (TQN) uses a Transformer-based architecture to attend to relevant video segments for each query, enabling query-specific response prediction.
• Each query is associated with a learnable query vector that attends over the entire video through multi-head self-attention, focusing on relevant temporal segments.
• A stochastic feature bank is introduced to handle dense sampling: only a random subset of features from densely sampled clips are computed and backpropagated per training step.
• The feature bank is updated stochastically during training, allowing the model to learn from diverse temporal contexts without exhausting GPU memory.
• The model is trained end-to-end using only query-level labels, without requiring bounding boxes or frame-level annotations.
• The approach supports both short-duration actions (e.g., 0.3 seconds) and longer sequences by attending over the full video duration with fine-grained temporal sensitivity.

Experimental results

Research questions

• RQ1 Can a query-response framework with learnable attention mechanisms effectively localize and classify fine-grained actions in untrimmed videos?
• RQ2 How can a deep learning model be trained efficiently on long, untrimmed videos with dense temporal sampling, given GPU memory limitations?
• RQ3 To what extent can a model achieve state-of-the-art performance in fine-grained action recognition using only RGB features, without optical flow or 3D convolutions?
• RQ4 How does the stochastic feature bank update strategy compare to standard training in terms of convergence and performance on long video sequences?
• RQ5 What is the impact of extended and dense temporal context on modeling subtle action differences in fine-grained video understanding?

Key findings

• The TQN model achieves state-of-the-art performance on the FineGym dataset, surpassing previous methods using only RGB features.
• On the Diving48 benchmark, the TQN with the stochastic feature bank outperforms existing approaches, even when they use additional supervision or optical flow.
• The ablation study confirms that the stochastic feature bank is critical for training stability and performance on long videos with dense sampling.
• The model demonstrates strong generalization to short, subtle actions (e.g., 0.3 seconds), showing that temporal attention can preserve fine-grained signals.
• The query-based mechanism enables better disambiguation of actions with similar visual appearances but different temporal structures.
• The method achieves high accuracy without relying on object or background cues, making it robust to the challenges of fine-grained action recognition.