[Paper Review] TFCNet: Temporal Fully Connected Networks for Static Unbiased Temporal Reasoning

Shiwen Zhang|arXiv (Cornell University)|Mar 11, 2022

Human Pose and Action Recognition4 citations

TL;DR

TFCNet introduces Temporal Fully Connected Blocks (TFC Blocks) that approximate fully connected layers along the temporal dimension to enhance global temporal receptive fields in 3D CNNs, significantly improving performance on static-unbiased temporal reasoning benchmarks. The method achieves state-of-the-art results on CATER (98.4% top-1 accuracy) and Diving48 (88.3% top-1 accuracy), outperforming RNNs, vision transformers, and prior CNNs by large margins.

ABSTRACT

Temporal Reasoning is one important functionality for vision intelligence. In computer vision research community, temporal reasoning is usually studied in the form of video classification, for which many state-of-the-art Neural Network structures and dataset benchmarks are proposed in recent years, especially 3D CNNs and Kinetics. However, some recent works found that current video classification benchmarks contain strong biases towards static features, thus cannot accurately reflect the temporal modeling ability. New video classification benchmarks aiming to eliminate static biases are proposed, with experiments on these new benchmarks showing that the current clip-based 3D CNNs are outperformed by RNN structures and recent video transformers. In this paper, we find that 3D CNNs and their efficient depthwise variants, when video-level sampling strategy is used, are actually able to beat RNNs and recent vision transformers by significant margins on static-unbiased temporal reasoning benchmarks. Further, we propose Temporal Fully Connected Block (TFC Block), an efficient and effective component, which approximates fully connected layers along temporal dimension to obtain video-level receptive field, enhancing the spatiotemporal reasoning ability. With TFC blocks inserted into Video-level 3D CNNs (V3D), our proposed TFCNets establish new state-of-the-art results on synthetic temporal reasoning benchmark, CATER, and real world static-unbiased dataset, Diving48, surpassing all previous methods.

Motivation & Objective

• To address the limitations of clip-based 3D CNNs on static-unbiased temporal reasoning benchmarks where models fail due to insufficient global temporal modeling.
• To investigate whether video-level sampling strategies can restore the competitive edge of 3D CNNs over RNNs and vision transformers on unbiased benchmarks.
• To design a novel, efficient component—TFC Blocks—that enables global temporal modeling in 3D CNNs without excessive parameter or FLOP growth.
• To establish a new state-of-the-art for video-level temporal reasoning on CATER and Diving48 using a lightweight, end-to-end trainable architecture.

Proposed method

• Proposes Temporal Fully Connected Blocks (TFC Blocks), which apply fully connected operations across the temporal dimension to model long-range dependencies in video features.
• Integrates TFC Blocks into Video-level 3D CNNs (V3D) and their depthwise variants to form TFCNets, enabling global temporal context learning at intermediate layers.
• Uses video-level sampling strategy during training and inference, ensuring full video context is used for feature learning, unlike clip-based sampling.
• Designs TFC Blocks with quadratic parameter scaling in temporal length but minimal FLOP overhead, making them computationally efficient.
• Employs a residual bottleneck architecture with TFC Blocks inserted after spatial convolutions to maintain spatial and temporal feature resolution.
• Compares TFC Blocks with SE and Nonlocal Blocks in ablation studies, demonstrating superior accuracy with lower parameter and FLOP cost.

Experimental results

Research questions

• RQ1 Can 3D CNNs with video-level sampling outperform RNNs and vision transformers on static-unbiased temporal reasoning benchmarks?
• RQ2 Does introducing a global temporal modeling mechanism like TFC Blocks significantly improve the performance of 3D CNNs on benchmarks like CATER and Diving48?
• RQ3 How does the TFC Block compare to existing global receptive field mechanisms such as SE and Nonlocal Blocks in terms of accuracy, parameters, and FLOPs?
• RQ4 Does the effectiveness of TFC Blocks depend on input sequence length, and if so, under what conditions is the gain most pronounced?
• RQ5 Can TFCNets achieve state-of-the-art performance without relying on external data or annotations?

Key findings

• TFCNets achieve 98.4% top-1 accuracy on the CATER benchmark, surpassing all prior methods, including vision transformers and models using external annotations.
• On Diving48 V2, TFCNets achieve 88.3% top-1 accuracy, which is 7.3% higher than the best-reported vision transformer and 11.6% higher than the prior SOTA on Diving48 V1.
• The video-level sampling strategy outperforms both clip-level and hybrid V4D sampling by over 30% absolute accuracy, confirming its superiority for global reasoning.
• TFC Blocks provide a 27 GFLOP reduction compared to Nonlocal Blocks while achieving higher accuracy, demonstrating superior efficiency and effectiveness.
• SE Blocks degrade performance on V3D, indicating that global average pooling disrupts fine-grained spatiotemporal structures, unlike TFC Blocks which preserve them.
• The performance gain of TFCNets is not due to increased parameters, as TFCNets have only 0.3 GFLOPs more than baseline V3D, yet achieve significant accuracy gains.