[Paper Review] Consensus-based Sequence Training for Video Captioning

Sang Phan, Gustav Eje Henter+2 more|arXiv (Cornell University)|Dec 27, 2017

Multimodal Machine Learning ApplicationsComputer Science33 references19 citations

TL;DR

This paper proposes Consensus-based Sequence Training (CST), a two-stage method that first uses a weighted cross-entropy pre-training step to reduce objective mismatch by treating ground-truth captions as model samples, then fine-tunes with REINFORCE using consensus scores among ground-truth captions as a fast, stable baseline. CST achieves a new state-of-the-art CIDEr score of 54.2 on MSRVTT, outperforming prior methods by over 2.9 points.

ABSTRACT

Captioning models are typically trained using the cross-entropy loss. However, their performance is evaluated on other metrics designed to better correlate with human assessments. Recently, it has been shown that reinforcement learning (RL) can directly optimize these metrics in tasks such as captioning. However, this is computationally costly and requires specifying a baseline reward at each step to make training converge. We propose a fast approach to optimize one's objective of interest through the REINFORCE algorithm. First we show that, by replacing model samples with ground-truth sentences, RL training can be seen as a form of weighted cross-entropy loss, giving a fast, RL-based pre-training algorithm. Second, we propose to use the consensus among ground-truth captions of the same video as the baseline reward. This can be computed very efficiently. We call the complete proposal Consensus-based Sequence Training (CST). Applied to the MSRVTT video captioning benchmark, our proposals train significantly faster than comparable methods and establish a new state-of-the-art on the task, improving the CIDEr score from 47.3 to 54.2.

Motivation & Objective

• To address the objective mismatch problem in video captioning, where cross-entropy training does not align with human-evaluated metrics like CIDEr.
• To reduce training instability and computational cost in reinforcement learning-based captioning by replacing greedy baseline estimation with consensus-based rewards.
• To enable efficient, stable pre-training that improves generalization before fine-tuning with RL, separating objective mismatch from exposure bias.
• To establish a new state-of-the-art on the MSRVTT benchmark by directly optimizing for consensus-based metrics like CIDEr.

Proposed method

• Proposes a weighted cross-entropy pre-training scheme (WXE) that treats ground-truth captions as model samples, effectively approximating REINFORCE with true labels.
• Introduces a self-consensus baseline (SCB) that computes the average CIDEr score among all ground-truth captions for a video as the baseline reward in REINFORCE.
• Uses the SCB to accelerate training by eliminating the need for greedy sampling during baseline estimation, reducing training time by ~2x compared to greedy baselines.
• Applies a two-stage training pipeline: first WXE pre-training to mitigate objective mismatch, then full REINFORCE fine-tuning with SCB to address exposure bias.
• Employs the REINFORCE algorithm with a baseline to stabilize policy gradient estimation, using CIDEr as the reward signal to optimize for human-like captioning.
• Extends the method to use sampled captions during training for the SCB, maintaining computational efficiency while allowing dynamic baseline estimation.

Experimental results

Research questions

• RQ1 Can replacing model-generated samples with ground-truth captions in REINFORCE training yield a stable, fast pre-training method that reduces objective mismatch?
• RQ2 Does using the consensus score among ground-truth captions as a baseline in REINFORCE training lead to faster convergence and better performance than greedy baseline estimation?
• RQ3 Can a two-stage training scheme—pre-training with weighted cross-entropy and fine-tuning with consensus-based REINFORCE—simultaneously address objective mismatch and exposure bias?
• RQ4 Does the proposed Consensus-based Sequence Training (CST) method achieve state-of-the-art performance on the MSRVTT video captioning benchmark when optimizing for CIDEr?

Key findings

• CST achieves a new state-of-the-art CIDEr score of 54.2 on the MSRVTT benchmark, significantly improving upon the previous best score of 51.7.
• The WXE pre-training stage alone improves the CIDEr score by over 2 points compared to standard cross-entropy training (from 47.3 to 49.7).
• Fine-tuning with the self-consensus baseline (SCB) improves performance by an additional 3.9 points, reaching 53.6, and further optimization using sampled captions achieves 54.2.
• The SCB baseline enables training that is approximately twice as fast as using a greedy baseline estimator, due to avoiding repeated CIDEr computation on greedy outputs.
• CST outperforms all prior methods on all metrics, with the largest gains observed on CIDEr, which correlates best with human judgment.
• Qualitative analysis shows CST generates more consistent and human-like captions, especially on videos with high variance in human-annotated captions, due to reduced sensitivity to outliers.