[Paper Review] RUC+CMU: System Report for Dense Captioning Events in Videos

Shizhe Chen, Yuqing Song+4 more|arXiv (Cornell University)|Jun 22, 2018

Multimodal Machine Learning Applications8 references4 citations

TL;DR

This paper presents a two-stage dense video captioning system combining temporal proposal ranking and ensemble caption generation, leveraging multimodal features (RGB, flow, audio) and bidirectional LSTM context modeling. It achieves state-of-the-art performance with an 8.529 METEOR score on the ActivityNet test set by fusing proposal scores and caption scores for re-ranking.

ABSTRACT

This notebook paper presents our system in the ActivityNet Dense Captioning in Video task (task 3). Temporal proposal generation and caption generation are both important to the dense captioning task. Therefore, we propose a proposal ranking model to employ a set of effective feature representations for proposal generation, and ensemble a series of caption models enhanced with context information to generate captions robustly on predicted proposals. Our approach achieves the state-of-the-art performance on the dense video captioning task with 8.529 METEOR score on the challenge testing set.

Motivation & Objective

• To address the challenge of dense video captioning by jointly optimizing temporal event localization and accurate caption generation.
• To improve proposal quality by incorporating context-aware features and boundary variance for more precise event localization.
• To enhance caption robustness by fusing multiple caption models enhanced with context and topic information.
• To achieve state-of-the-art performance on the ActivityNet Dense Captioning benchmark through systematic model ensembling and re-ranking.

Proposed method

• Divides videos into 64-frame non-overlapping segments and extracts multimodal features: ResNet (image), I3D (motion), and VGGish (audio).
• Uses a bidirectional LSTM to encode bidirectional context from segment features, trained to predict ground-truth caption concepts.
• Generates candidate proposals via a sliding window with multiple window lengths derived from ground-truth distribution clusters.
• Employs a two-layer feed-forward neural network to rank proposals using four features: internal (proposal content), external (context), boundary variance, and location/duration proportions.
• Ensembles three caption models: vanilla LSTM, temporal attention-based, and topic-guided (using 200 ActivityNet categories), all fine-tuned with self-critical reinforcement learning.
• Re-ranks final outputs using a product of proposal score and caption score (s = s_p × s_c), selecting top 10 captions per video.

Experimental results

Research questions

• RQ1 How can multimodal and contextual features improve temporal proposal generation in dense video captioning?
• RQ2 To what extent does ensemble caption modeling with context and topic guidance enhance caption quality and robustness?
• RQ3 Can a learned proposal ranking model effectively filter low-quality proposals while preserving high recall?
• RQ4 How does the integration of proposal and caption quality scores through re-ranking affect overall performance on dense captioning?

Key findings

• The sliding window baseline generates 241 proposals per video with 28% precision (tiou ≥ 0.3), but recall is high at 98%.
• After applying the proposal ranking model (threshold s_p > 0.5), average proposal count drops to 53, with precision improving to 71% (tiou ≥ 0.3) and recall remaining at 84%.
• The caption ensemble model achieves 13.75 METEOR on ground-truth proposals, outperforming individual models, demonstrating the benefit of model fusion.
• On predicted proposals, the ensemble model maintains strong robustness, achieving 12.44 METEOR, a drop from 13.75 due to imperfect proposals and increased proposal count.
• The final system achieves a state-of-the-art 8.529 METEOR score on the ActivityNet test set, indicating strong generalization and performance.