[Paper Review] Structured Context Transformer for Generic Event Boundary Detection

Congcong Li, Xinyao Wang+5 more|arXiv (Cornell University)|Jun 7, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes Structured Context Transformer (SC-Transformer), an end-to-end method for generic event boundary detection (GEBD) that uses structured sequence partitioning to enable linear-complexity temporal modeling via Transformers. It computes group similarities between frame representations and employs a lightweight FCN for boundary prediction, achieving state-of-the-art performance on Kinetics-GEBD and TAPOS with 1.3% absolute F1 improvement over DDM-Net.

ABSTRACT

Generic Event Boundary Detection (GEBD) aims to detect moments where humans naturally perceive as event boundaries. In this paper, we present Structured Context Transformer (or SC-Transformer) to solve the GEBD task, which can be trained in an end-to-end fashion. Specifically, we use the backbone convolutional neural network (CNN) to extract the features of each video frame. To capture temporal context information of each frame, we design the structure context transformer (SC-Transformer) by re-partitioning input frame sequence. Note that, the overall computation complexity of SC-Transformer is linear to the video length. After that, the group similarities are computed to capture the differences between frames. Then, a lightweight fully convolutional network is used to determine the event boundaries based on the grouped similarity maps. To remedy the ambiguities of boundary annotations, the Gaussian kernel is adopted to preprocess the ground-truth event boundaries to further boost the accuracy. Extensive experiments conducted on the challenging Kinetics-GEBD and TAPOS datasets demonstrate the effectiveness of the proposed method compared to the state-of-the-art methods.

Motivation & Objective

• Address the challenge of detecting event boundaries in videos with minimal human-level granularity, where boundaries are taxonomy-free and perception-based.
• Overcome the limitations of existing methods that suffer from quadratic computation complexity or redundant frame processing due to lack of structured context modeling.
• Improve boundary detection accuracy by modeling temporal context efficiently while handling the inherent ambiguity in human-annotated event boundaries.
• Enable end-to-end training with high inference efficiency by designing a linear-complexity architecture suitable for long video sequences.
• Introduce a novel group similarity mechanism to enhance discriminative feature learning for boundary detection, distinct from prior global similarity or single-group approaches.

Proposed method

• Use a CNN backbone to extract 2D features from each video frame, followed by global average pooling to form a sequence of frame embeddings.
• Apply the Structured Partition of Sequence (SPoS) mechanism to re-partition the frame sequence into local, structured contexts for each frame, enabling localized attention with linear complexity.
• Employ Transformer encoder blocks on each structured context to learn high-level representations with efficient, scalable self-attention over local sequences.
• Compute group similarities between frame representations across multiple channels to capture discriminative differences, using a learnable grouping mechanism that improves feature discrimination.
• Use a lightweight fully convolutional network (FCN) to predict event boundaries from the grouped similarity maps, enabling dense, frame-wise boundary classification.
• Apply Gaussian kernel smoothing to ground-truth annotations during training to reduce label ambiguity and improve model convergence and robustness.

Experimental results

Research questions

• RQ1 Can a structured context modeling approach with linear complexity outperform existing global or local self-attention mechanisms in GEBD?
• RQ2 How does group similarity learning compare to single-group or global similarity computation in capturing discriminative frame differences for boundary detection?
• RQ3 To what extent does Gaussian smoothing of ground-truth boundaries improve training stability and model generalization in ambiguous GEBD annotations?
• RQ4 What is the optimal window size for structured context partitioning in terms of capturing sufficient temporal context without introducing noise?
• RQ5 How does model width and number of similarity groups affect performance and efficiency in the proposed SC-Transformer architecture?

Key findings

• The proposed SC-Transformer achieves a 1.3% absolute F1 improvement over DDM-Net on the Kinetics-GEBD dataset, demonstrating state-of-the-art performance.
• The method achieves an average F1 score of 0.881 on Kinetics-GEBD and 0.882 on TAPOS, with consistent gains across all evaluation thresholds (0.05, 0.25, 0.5).
• Using Gaussian smoothing with BCE loss improves performance by 0.016 average F1, confirming its effectiveness in mitigating label ambiguity and overconfidence.
• The optimal adjacent window size for context modeling is K=8, as larger values lead to performance saturation or slight degradation due to noise from multiple boundaries.
• Group similarity with G=8 achieves the highest performance (0.882 average F1), showing that multi-group similarity learning enhances discriminative feature learning.
• The model maintains linear computational complexity with respect to video length, enabling efficient inference and training, especially on long videos.