[Paper Review] Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs

Shengyu Feng, Subarna Tripathi+3 more|arXiv (Cornell University)|Dec 18, 2021

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes DSG-DETR, a Dynamic Scene Graph Detection Transformer that leverages long-term temporal dependencies through object-level tracklets and two dedicated transformers to model object consistency and relationship dynamics. It achieves state-of-the-art performance on Action Genome by significantly outperforming prior methods, approaching an upper-bound performance estimated using ground-truth object tracks.

ABSTRACT

Dynamic scene graph generation from a video is challenging due to the temporal dynamics of the scene and the inherent temporal fluctuations of predictions. We hypothesize that capturing long-term temporal dependencies is the key to effective generation of dynamic scene graphs. We propose to learn the long-term dependencies in a video by capturing the object-level consistency and inter-object relationship dynamics over object-level long-term tracklets using transformers. Experimental results demonstrate that our Dynamic Scene Graph Detection Transformer (DSG-DETR) outperforms state-of-the-art methods by a significant margin on the benchmark dataset Action Genome. Our ablation studies validate the effectiveness of each component of the proposed approach. The source code is available at https://github.com/Shengyu-Feng/DSG-DETR.

Motivation & Objective

• To investigate whether capturing long-term temporal dependencies improves dynamic scene graph generation from videos.
• To quantify the performance gain from long-term consistency using ground-truth object tracks as an upper-bound.
• To develop a method that achieves near-optimal performance without access to ground-truth tracks by leveraging predicted object tracklets.
• To demonstrate that modeling both object consistency and relationship transition dynamics over long sequences enhances scene graph prediction accuracy.

Proposed method

• Constructs object-level tracklets online using Hungarian matching on predicted detections across frames to ensure temporal consistency.
• Employs a transformer encoder with sinusoidal positional encoding to model long-term dependencies in object representations across tracklets.
• Uses a separate transformer encoder to model transitions in subject-object relationship classes over time, replacing STTran's temporal decoder with a temporal encoder.
• Integrates temporal position embeddings into the object transformer to enhance modeling of sequential object identities.
• Leverages both object and relationship transformers to jointly refine predictions using long-range temporal context.
• Trains end-to-end on the Action Genome dataset using a joint detection and scene graph generation loss.

Experimental results

Research questions

• RQ1 Can modeling long-term temporal dependencies in object and relationship sequences significantly improve dynamic scene graph generation?
• RQ2 How much performance gain is achievable by exploiting long-term consistency, as measured by an upper-bound using ground-truth object tracks?
• RQ3 Can a model trained on predicted tracklets achieve performance close to this upper-bound?
• RQ4 Does capturing both object consistency and relationship dynamics over long sequences lead to better generalization than short-term modeling?

Key findings

• DSG-DETR achieves state-of-the-art performance on the Action Genome benchmark, significantly outperforming existing methods in both SGCls and SGDet evaluation settings.
• The hypothetical best case using ground-truth object tracks shows a substantial performance gain over current methods, validating the importance of long-term dependencies.
• DSG-DETR approaches this upper-bound performance, demonstrating that long-term dependencies are the key to improved dynamic scene graph generation.
• Ablation studies show that the object transformer with positional encoding improves R@K by 0.2 points under constraint, and the relationship transformer with long-term modeling improves performance by 0.3 points over STTran.
• Long-term temporal context improves SGG performance by 3–6 points over baseline, while short-term context only improves by 3–4 points, showing the critical role of long-range modeling.
• Qualitative results show DSG-DETR correctly recovers misclassified objects (e.g., 'dish' as 'food') and maintains consistent predictions across occlusions and motion, unlike STTran and Faster R-CNN.