[Paper Review] Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding

Daizong Liu, Xiang Fang+2 more|arXiv (Cornell University)|Mar 6, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes MA3SRN, a novel 3D semantic reasoning network for temporal sentence grounding that integrates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object features through a motion-appearance associating module. The model achieves state-of-the-art performance on ActivityNet Caption, Charades-STA, and TACoS, with gains of up to 45.63% (R@1, IoU=0.7) and 73.86% (R@5, IoU=0.7) on Charades-STA.

ABSTRACT

Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query. Most previous works focus on learning frame-level features of each whole frame in the entire video, and directly match them with the textual information. Such frame-level feature extraction leads to the obstacles of these methods in distinguishing ambiguous video frames with complicated contents and subtle appearance differences, thus limiting their performance. In order to differentiate fine-grained appearance similarities among consecutive frames, some state-of-the-art methods additionally employ a detection model like Faster R-CNN to obtain detailed object-level features in each frame for filtering out the redundant background contents. However, these methods suffer from missing motion analysis since the object detection module in Faster R-CNN lacks temporal modeling. To alleviate the above limitations, in this paper, we propose a novel Motion- and Appearance-guided 3D Semantic Reasoning Network (MA3SRN), which incorporates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object-level features to better reason the spatial-temporal object relations for accurately modelling the activity among consecutive frames. Specifically, we first develop three individual branches for motion, appearance, and 3D encoding separately to learn fine-grained motion-guided, appearance-guided, and 3D-aware object features, respectively. Then, both motion and appearance information from corresponding branches are associated to enhance the 3D-aware features for the final precise grounding. Extensive experiments on three challenging datasets (ActivityNet Caption, Charades-STA and TACoS) demonstrate that the proposed MA3SRN model achieves a new state-of-the-art.

Motivation & Objective

• Address the limitations of frame-level feature extraction in distinguishing subtle visual differences and foreground-background content in temporally adjacent video frames.
• Overcome the lack of temporal modeling in detection-based methods that rely on Faster R-CNN, which fails to capture motion context at the object level.
• Improve temporal sentence grounding by jointly modeling motion, appearance, and 3D-aware spatio-temporal object relations for more accurate activity localization.
• Develop a unified framework that effectively fuses motion and appearance cues with 3D-aware features to enhance grounding precision.

Proposed method

• Design three dedicated branches: one for optical-flow-guided motion-aware features, one for detection-based appearance-aware features (using Faster R-CNN), and one for 3D-aware feature learning via 3D convolutional networks.
• Integrate motion and appearance information into the 3D-aware features through a novel motion-appearance associating module that enhances feature representation.
• Use optical flow to capture fine-grained motion dynamics between consecutive frames, enabling better discrimination of similar actions like 'open' and 'close'.
• Apply non-maximum suppression and confidence thresholding to control the number of detected objects per frame (K=20), reducing redundancy while preserving key instances.
• Train the model end-to-end with a contrastive loss and a boundary-aware regression head to predict start and end timestamps of the grounded segment.
• Optimize hyperparameters such as the positive threshold λ (optimal at 0.55) and object number K to balance precision and computational cost.

Experimental results

Research questions

• RQ1 Can integrating optical-flow-guided motion features improve the discrimination of semantically similar actions like 'open' and 'close' in video frames?
• RQ2 To what extent does incorporating detection-based appearance features enhance grounding accuracy in complex scenes with ambiguous foreground-background content?
• RQ3 How effective is the fusion of motion and appearance information with 3D-aware features in modeling spatio-temporal object relations for temporal grounding?
• RQ4 What is the optimal configuration of hyperparameters such as the positive threshold λ and the number of detected objects K per frame?
• RQ5 Does the proposed motion-appearance associating module significantly outperform direct concatenation of features in grounding performance?

Key findings

• The proposed MA3SRN achieves a new state-of-the-art on Charades-STA, with 45.63% (R@1, IoU=0.7) and 73.86% (R@5, IoU=0.7), surpassing previous methods.
• The ablation study shows that equipping 3D-aware features with both motion and appearance information yields a 1.79% and 1.81% improvement over the baseline in R@1 and R@5, respectively.
• The motion-appearance associating module improves performance by 1.79% in R@1 and 1.81% in R@5 compared to direct concatenation, demonstrating its effectiveness.
• The optimal positive threshold λ is 0.55, as performance degrades beyond this value due to over-strict filtering of positive proposals.
• Using K=20 detected objects per frame yields the best trade-off between performance and GPU memory cost, with K=30 offering only marginal gains at significantly higher cost.
• Qualitative results show that MA3SRN better captures fine-grained object differences (e.g., 'door' vs. 'closet') and complex actions (e.g., 'open' vs. 'hold'), outperforming frame-level baselines like DRN and CBLN.