[Paper Review] Video Captioning with Boundary-aware Hierarchical Language Decoding and Joint Video Prediction

Xiangxi Shi, Jianfei Cai+2 more|arXiv (Cornell University)|Jul 8, 2018

Multimodal Machine Learning Applications3 citations

TL;DR

This paper proposes a boundary-aware hierarchical language decoder for video captioning, integrating a high-level GRU for global sentence modeling and a low-level GRU with a binary gate for phrase-level boundary detection. The method achieves state-of-the-art performance on MSR-VTT and MSVD datasets by jointly optimizing video captioning and prediction with shared attention and boundary-aware encoding.

ABSTRACT

The explosion of video data on the internet requires effective and efficient technology to generate captions automatically for people who are not able to watch the videos. Despite the great progress of video captioning research, particularly on video feature encoding, the language decoder is still largely based on the prevailing RNN decoder such as LSTM, which tends to prefer the frequent word that aligns with the video. In this paper, we propose a boundary-aware hierarchical language decoder for video captioning, which consists of a high-level GRU based language decoder, working as a global (caption-level) language model, and a low-level GRU based language decoder, working as a local (phrase-level) language model. Most importantly, we introduce a binary gate into the low-level GRU language decoder to detect the language boundaries. Together with other advanced components including joint video prediction, shared soft attention, and boundary-aware video encoding, our integrated video captioning framework can discover hierarchical language information and distinguish the subject and the object in a sentence, which are usually confusing during the language generation. Extensive experiments on two widely-used video captioning datasets, MSR-Video-to-Text (MSR-VTT) \cite{xu2016msr} and YouTube-to-Text (MSVD) \cite{chen2011collecting} show that our method is highly competitive, compared with the state-of-the-art methods.

Motivation & Objective

• To address the limitation of RNN-based language decoders in video captioning that overfit to frequent word co-occurrences.
• To improve sentence-level coherence and reduce hallucinations by modeling hierarchical linguistic structure.
• To enhance video captioning performance through joint learning of video captioning and video prediction tasks.
• To detect linguistic boundaries (e.g., subject-verb or noun-verb transitions) using a binary gate in a low-level GRU decoder.
• To integrate boundary-aware video encoding with hierarchical language decoding for improved semantic alignment.

Proposed method

• Proposes a hierarchical language decoder with a high-level GRU (global language model) and a low-level GRU (local phrase-level model) for sentence generation.
• Introduces a Binary Gated Recurrent Unit (B-GRU) that uses a learnable binary gate to detect linguistic boundaries between words or phrases.
• Employs shared soft attention between video captioning and video prediction tasks to improve feature alignment and reduce parameters.
• Uses boundary-aware video encoding to improve temporal localization of visual events.
• Integrates joint video prediction as a multi-task learning objective to enhance feature representation and reasoning.
• Fuses all components into an end-to-end trainable framework with beam search inference.

Experimental results

Research questions

• RQ1 Can a hierarchical language decoder with boundary detection improve the semantic structure and accuracy of generated video captions?
• RQ2 How does joint video prediction enhance the performance of video captioning models?
• RQ3 To what extent does shared attention between captioning and prediction tasks improve feature learning?
• RQ4 Can boundary detection in the language decoder reduce common errors like subject-object confusion or incorrect word co-occurrences?
• RQ5 How does the proposed framework compare to state-of-the-art methods when trained only on a single dataset?

Key findings

• The proposed model achieves a BLEU-4 score of 50.3 and a CIDEr score of 74.3 on the MSR-VTT dataset, outperforming previous methods trained on a single dataset.
• On the MSVD dataset, the model achieves a BLEU-4 score of 39.8 and a CIDEr score of 43.3, demonstrating strong performance across metrics.
• The ablation study confirms that each component—boundary detection, shared attention, video prediction, and hierarchical language modeling—contributes to performance gains.
• The binary gate in the B-GRU successfully detects linguistic boundaries, such as between noun phrases and verbs, as shown in qualitative examples.
• The shared attention mechanism outperforms separate attention modules for captioning and prediction, proving its effectiveness in feature sharing.
• Qualitative results show reduced hallucinations, such as avoiding captions like 'a woman is riding a woman', and improved subject-verb alignment.