[Paper Review] Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs

Kaifeng Gao, Long Chen+3 more|arXiv (Cornell University)|Dec 8, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a novel classification-then-grounding framework for Video Scene Graph Generation (VidSGG), reformulating video scene graphs as temporal bipartite graphs to overcome limitations of proposal-based methods. By classifying predicates on full subject-object tracklets before grounding temporal spans, the method preserves high-order relations, avoids inconsistent labeling, and improves performance, achieving state-of-the-art results on VidOR and VidVRD with a new model, BIG.

ABSTRACT

Today's VidSGG models are all proposal-based methods, i.e., they first generate numerous paired subject-object snippets as proposals, and then conduct predicate classification for each proposal. In this paper, we argue that this prevalent proposal-based framework has three inherent drawbacks: 1) The ground-truth predicate labels for proposals are partially correct. 2) They break the high-order relations among different predicate instances of a same subject-object pair. 3) VidSGG performance is upper-bounded by the quality of the proposals. To this end, we propose a new classification-then-grounding framework for VidSGG, which can avoid all the three overlooked drawbacks. Meanwhile, under this framework, we reformulate the video scene graphs as temporal bipartite graphs, where the entities and predicates are two types of nodes with time slots, and the edges denote different semantic roles between these nodes. This formulation takes full advantage of our new framework. Accordingly, we further propose a novel BIpartite Graph based SGG model: BIG. It consists of a classification stage and a grounding stage, where the former aims to classify the categories of all the nodes and the edges, and the latter tries to localize the temporal location of each relation instance. Extensive ablations on two VidSGG datasets have attested to the effectiveness of our framework and BIG. Code is available at https://github.com/Dawn-LX/VidSGG-BIG.

Motivation & Objective

• Address inherent drawbacks in proposal-based VidSGG methods, including inconsistent ground-truth labeling, broken high-order relations, and proposal quality bottlenecks.
• Reformulate video scene graphs as temporal bipartite graphs to model entities and predicates with time-aware node types and semantic role edges.
• Develop a two-stage framework—classification followed by grounding—to improve predicate classification and temporal localization accuracy.
• Enable multi-instance grounding for subject-object pairs with multiple predicate instances, enhancing robustness to complex video interactions.
• Achieve state-of-the-art performance on VidOR and VidVRD benchmarks by leveraging full tracklet context and preserving relational structure.

Proposed method

• Reformulate VidSGG as a temporal bipartite graph with two node types: entities (subjects/objects) and predicates, each associated with time slots.
• Model semantic roles between entities and predicates using directed edges in the bipartite graph, capturing temporal and relational structure.
• Decompose the task into two stages: first, classify all node and edge categories using global tracklet representations; second, localize temporal spans of each predicted relation instance.
• Use multi-instance grounding with score-based filtering and temporal NMS to predict multiple time slots per relation triplet, improving robustness.
• Employ a two-stage training pipeline: train classification and grounding stages separately with optimized hyperparameters and loss weighting.
• Apply label assignment that preserves multiple ground-truth instances per subject-object pair, with 98.43% of bins containing only one instance, validating the scheme’s suitability.

Figure 1 : (a) : The pipeline of proposal-based framework. Given a video, it first generates numerous proposals (with different time slots), and then conducts predicate classification for each proposal. (b) : The pipeline of classification-then-grounding framework. It first conducts predicate classi

Experimental results

Research questions

• RQ1 Can a classification-then-grounding framework outperform proposal-based VidSGG methods by avoiding inconsistent and partial ground-truth labeling?
• RQ2 How does modeling video scene graphs as temporal bipartite graphs improve the modeling of high-order relations between multiple predicate instances on the same subject-object pair?
• RQ3 To what extent does VidSGG performance depend on proposal quality, and can a proposal-free classification stage mitigate this upper bound?
• RQ4 How effective is multi-instance grounding in capturing multiple simultaneous or sequential relations between the same subject-object pair?
• RQ5 Can the proposed framework achieve state-of-the-art performance on standard VidSGG benchmarks like VidOR and VidVRD?

Key findings

• The proposed classification-then-grounding framework outperforms existing proposal-based methods on both VidOR and VidVRD, demonstrating superior generalization and robustness.
• On VidOR, the model achieves a new state-of-the-art performance with a 1.5% absolute improvement in mean mAP over prior methods, validating the effectiveness of the new framework.
• On VidVRD, the model achieves a 2.1% absolute gain in mean mAP, showing consistent gains across different datasets and settings.
• Ablation studies confirm that the framework significantly reduces label inconsistency and preserves high-order relational context, which is lost in proposal-based methods.
• The multi-instance grounding mechanism effectively captures 32% of predicates with multiple ground-truth instances, and the label assignment scheme is highly effective, with only 1.57% of bins containing two or more instances.
• The model’s inference pipeline, including score filtering and temporal NMS, improves prediction reliability, reducing false positives and enhancing localization accuracy.

Figure 2 : Left : A video example and its ground-truth visual relation triplets. Right : The corresponding temporal bipartite graph. Comparisons with existing formulation are left in appendix.