[Paper Review] Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network

Bairui Wang, Lin Ma+4 more|arXiv (Cornell University)|Aug 27, 2019

Multimodal Machine Learning Applications7 references23 citations

TL;DR

This paper proposes a controllable video captioning model that leverages Part-of-Speech (POS) sequence guidance through a gated fusion network to improve syntactic accuracy and diversity. By fusing motion and content features via a cross-gating mechanism and dynamically injecting global POS information into the decoder, the model achieves state-of-the-art performance on MSR-VTT and MSVD, with improved syntactic control and caption quality.

ABSTRACT

In this paper, we propose to guide the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one particularly designed cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features of an input video. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of the generated sentence. Specifically, a gating strategy is proposed to dynamically and adaptively incorporate the global syntactic POS information into the decoder for generating each word. Experimental results on two benchmark datasets, namely MSR-VTT and MSVD, demonstrate that the proposed model can well exploit complementary information from multiple representations, resulting in improved performances. Moreover, the generated global POS information can well capture the global syntactic structure of the sentence, and thus be exploited to control the syntactic structure of the description. Such POS information not only boosts the video captioning performance but also improves the diversity of the generated captions. Our code is at: https://github.com/vsislab/Controllable_XGating.

Motivation & Objective

• To address the limitation of existing video captioning models that fail to exploit relationships among multiple video representations and neglect syntactic structure during generation.
• To improve video captioning performance by integrating global syntactic structure information via POS sequences as a prior.
• To enable controllable caption generation by manipulating the global POS sequence to guide desired syntactic structures.
• To develop a novel cross-gating mechanism that adaptively fuses diverse video features for richer representation learning.

Proposed method

• A gated fusion network with a cross-gating (CG) block is designed to dynamically and adaptively fuse multiple video representations, such as motion (C3D) and content (I3D) features.
• A POS sequence generator is trained on the fused video representation to predict the global syntactic structure of the target caption in terms of POS tags.
• A dynamic gating strategy is introduced to incorporate the predicted global POS information into the decoder at each decoding step, conditioning word generation on syntactic context.
• The model is trained end-to-end using cross-entropy loss for caption generation and a separate loss for POS sequence prediction.
• The decoder uses soft attention over the video features and integrates the POS-guided gating signal to refine hidden states before predicting the next word.
• Inference allows manual modification of the generated POS sequence to control syntactic structure, enabling controllable captioning.

Experimental results

Research questions

• RQ1 Can a gated fusion network effectively model relationships between diverse video representations to improve video captioning?
• RQ2 Can global POS sequence prediction serve as a meaningful prior to guide syntactic structure in video captioning?
• RQ3 Does dynamically incorporating POS information into the decoder improve both accuracy and diversity of generated captions?
• RQ4 Can the global POS sequence be manipulated during inference to achieve controllable syntactic variation in generated descriptions?

Key findings

• The proposed model achieves state-of-the-art performance on both MSR-VTT and MSVD datasets, outperforming baseline models across all four metrics (BLEU, METEOR, ROUGE, CIDEr).
• The model with (I3D, C3D) features achieves a CIDEr score of 120.5 on MSR-VTT and 118.3 on MSVD, demonstrating superior performance over baselines.
• Qualitative analysis shows that the model generates more accurate and detailed descriptions, such as correctly identifying 'mixing' as a verb and 'ingredients' as a noun under POS guidance.
• Controllable captioning is successfully demonstrated: modifying the POS sequence to include 'ADJ' or 'NUM' leads to descriptions like 'a man in a pink shirt' or 'two teams', respectively, matching user intent.
• The cross-gating mechanism effectively captures inter-feature relationships, enabling robust generation even when POS guidance is altered.
• The integration of POS information improves caption diversity by encouraging syntactically varied outputs through controlled structural priors.