[Paper Review] Learning to Discretely Compose Reasoning Module Networks for Video Captioning

Ganchao Tan, Daqing Liu+2 more|arXiv (Cornell University)|Jul 17, 2020

Multimodal Machine Learning Applications36 references6 citations

TL;DR

This paper proposes Reasoning Module Networks (RMN), a novel video captioning framework that dynamically composes three spatio-temporal reasoning modules—Locate, Relate, and Func—using a discrete, Gumbel-approximated module selector trained with a linguistic loss. RMN achieves state-of-the-art performance on MSVD and MSR-VTT with explicit, explainable reasoning steps and improved CIDEr scores of 94.4 and 49.6, respectively.

ABSTRACT

Generating natural language descriptions for videos, i.e., video captioning, essentially requires step-by-step reasoning along the generation process. For example, to generate the sentence "a man is shooting a basketball", we need to first locate and describe the subject "man", next reason out the man is "shooting", then describe the object "basketball" of shooting. However, existing visual reasoning methods designed for visual question answering are not appropriate to video captioning, for it requires more complex visual reasoning on videos over both space and time, and dynamic module composition along the generation process. In this paper, we propose a novel visual reasoning approach for video captioning, named Reasoning Module Networks (RMN), to equip the existing encoder-decoder framework with the above reasoning capacity. Specifically, our RMN employs 1) three sophisticated spatio-temporal reasoning modules, and 2) a dynamic and discrete module selector trained by a linguistic loss with a Gumbel approximation. Extensive experiments on MSVD and MSR-VTT datasets demonstrate the proposed RMN outperforms the state-of-the-art methods while providing an explicit and explainable generation process. Our code is available at https://github.com/tgc1997/RMN.

Motivation & Objective

• To address the lack of explicit, step-by-step visual reasoning in existing video captioning models.
• To enable dynamic and discrete composition of reasoning modules during caption generation.
• To improve caption quality by integrating linguistic structure (e.g., part-of-speech tags) into the reasoning process.
• To design specialized spatio-temporal reasoning modules for video understanding across space and time.
• To achieve state-of-the-art performance with an interpretable and explainable generation process.

Proposed method

• The model employs three dedicated spatio-temporal reasoning modules: Locate (spatio-temporal attention for visual word generation), Relate (pairwise region interaction for action word generation), and Func (language context-aware function word generation).
• A discrete module selector dynamically chooses one of the three modules at each decoding step using Gumbel-Softmax to enable differentiable sampling.
• The module selector is trained with a linguistic loss based on part-of-speech (POS) tag supervision to align module choices with grammatical structure.
• The framework is end-to-end trainable using a combination of cross-entropy loss for captioning and linguistic loss for module selection.
• The model uses a standard encoder-decoder architecture with a CNN backbone for video feature extraction and an RNN decoder for autoregressive caption generation.
• The Gumbel-approximation enables backpropagation through discrete module selections, allowing joint optimization of modules and selector.

Experimental results

Research questions

• RQ1 Can a dynamic, discrete module composition mechanism improve reasoning capacity in video captioning?
• RQ2 How can linguistic structure (e.g., POS tags) be effectively integrated to guide module selection during caption generation?
• RQ3 Do specialized spatio-temporal reasoning modules (Locate, Relate, Func) lead to more interpretable and accurate captioning?
• RQ4 Can a Gumbel-approximated discrete selector enable end-to-end training of a reasoning module network?
• RQ5 Does incorporating linguistic supervision improve both module selection accuracy and overall caption quality?

Key findings

• RMN achieves new state-of-the-art performance on both MSVD and MSR-VTT datasets, with a CIDEr score of 94.4 on MSR-VTT and 92.8 on MSVD.
• The linguistic loss consistently improves performance across all metrics, demonstrating its effectiveness in aligning module selection with grammatical structure.
• Word cloud analysis confirms that each module specializes in its intended function: Locate for visual words (e.g., 'man', 'basketball'), Relate for action words (e.g., 'shooting', 'playing'), and Func for function words (e.g., 'a', 'is').
• The model's generation process is explicit and explainable, as each word is generated by a clearly defined reasoning module at each step.
• RMN (H+L) outperforms RMN (S+L), indicating that hard sampling with linguistic supervision yields better results than soft fusion.
• The ablation study confirms that the linguistic loss significantly improves module selection accuracy and overall caption quality.