[Paper Review] TF-Blender: Temporal Feature Blender for Video Object Detection

Yiming Cui, Liqi Yan+2 more|arXiv (Cornell University)|Aug 12, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

TF-Blender is a general, plug-in framework for video object detection that enhances feature representation by modeling fine-grained temporal relations between all frame pairs in a video clip. It uses a temporal relation module to compute adaptive weights, a feature adjustment module to preserve spatial details, and a feature blender to fuse enriched features, achieving +0.7% mAP on ImageNet VID and +1.5% mAP on YouTube-VIS over SOTA methods.

ABSTRACT

Video objection detection is a challenging task because isolated video frames may encounter appearance deterioration, which introduces great confusion for detection. One of the popular solutions is to exploit the temporal information and enhance per-frame representation through aggregating features from neighboring frames. Despite achieving improvements in detection, existing methods focus on the selection of higher-level video frames for aggregation rather than modeling lower-level temporal relations to increase the feature representation. To address this limitation, we propose a novel solution named TF-Blender,which includes three modules: 1) Temporal relation mod-els the relations between the current frame and its neighboring frames to preserve spatial information. 2). Feature adjustment enriches the representation of every neigh-boring feature map; 3) Feature blender combines outputs from the first two modules and produces stronger features for the later detection tasks. For its simplicity, TF-Blender can be effortlessly plugged into any detection network to improve detection behavior. Extensive evaluations on ImageNet VID and YouTube-VIS benchmarks indicate the performance guarantees of using TF-Blender on recent state-of-the-art methods.

Motivation & Objective

• To address feature degradation in video object detection caused by motion blur, occlusion, and appearance variation.
• To improve feature representation by modeling both local spatial consistency and global semantic correspondence across video frames.
• To design a general-purpose module that enhances any detection network with minimal architectural changes.
• To overcome limitations of prior methods that focus only on global relations between current and neighboring frames or use fixed frame aggregation.
• To enable end-to-end, robust feature aggregation that preserves spatial structure while enhancing temporal coherence.

Proposed method

• The temporal relation module computes adaptive weights between every pair of frames in a clip using a learnable function g(fi,fj), capturing both local and global relations.
• The feature adjustment module enhances neighboring features by aggregating them with other neighbors, adding spatial constraints to preserve fine-grained details.
• The feature blender combines outputs from the temporal relation and feature adjustment modules using learned attention weights to generate stronger, more robust features.
• The framework is implemented as a lightweight module that can be inserted into any detection backbone, such as ResNeXt or Swin Transformer, without retraining the main network.
• The method uses a multi-head cross-attention mechanism to compute similarity scores between features of different frames, enabling dynamic, context-aware aggregation.
• The entire module is differentiable and trained end-to-end with the detection head, ensuring joint optimization of feature learning and detection.

Experimental results

Research questions

• RQ1 Can modeling pairwise temporal relations between all frames in a video clip improve feature representation for object detection?
• RQ2 Does preserving local spatial structure during feature aggregation lead to better detection performance under appearance variations and motion blur?
• RQ3 Can a general-purpose module be designed to enhance any existing video object detector with minimal architectural changes?
• RQ4 How does the proposed method compare in accuracy and efficiency to existing feature aggregation methods like FGFA and SELSA?
• RQ5 What is the impact of object size and motion speed on the performance gain of the proposed method?

Key findings

• TF-Blender achieves an absolute mAP gain of 0.7% on the ImageNet VID benchmark when integrated with state-of-the-art detectors.
• On the YouTube-VIS benchmark, the method improves mAP by 1.5% over SOTA methods, demonstrating strong generalization across datasets.
• The method shows the highest performance gain on large objects, with improved mAP due to higher-resolution feature maps benefiting from fine-grained weight learning.
• The largest improvement is observed on slowly moving objects (37.9% of test set), suggesting better handling of motion ambiguity and appearance variation.
• The method maintains a favorable speed-accuracy tradeoff, with only a marginal increase in inference cost due to the lightweight design of the TF-Blender module.
• Ablation studies confirm that increasing the number of layers in the mini-network leads to overfitting, and a 3-layer structure with 3×3 kernels yields optimal performance.