[Paper Review] End-to-End Dense Video Captioning with Masked Transformer

Luowei Zhou, Yingbo Zhou+3 more|arXiv (Cornell University)|Apr 3, 2018

Multimodal Machine Learning Applications39 references42 citations

TL;DR

Presents an end-to-end transformer model for dense video captioning that jointly learns event proposals and captions using a differentiable mask to align proposal and captioning, leveraging self-attention instead of RNNs.

ABSTRACT

Dense video captioning aims to generate text descriptions for all events in an untrimmed video. This involves both detecting and describing events. Therefore, all previous methods on dense video captioning tackle this problem by building two models, i.e. an event proposal and a captioning model, for these two sub-problems. The models are either trained separately or in alternation. This prevents direct influence of the language description to the event proposal, which is important for generating accurate descriptions. To address this problem, we propose an end-to-end transformer model for dense video captioning. The encoder encodes the video into appropriate representations. The proposal decoder decodes from the encoding with different anchors to form video event proposals. The captioning decoder employs a masking network to restrict its attention to the proposal event over the encoding feature. This masking network converts the event proposal to a differentiable mask, which ensures the consistency between the proposal and captioning during training. In addition, our model employs a self-attention mechanism, which enables the use of efficient non-recurrent structure during encoding and leads to performance improvements. We demonstrate the effectiveness of this end-to-end model on ActivityNet Captions and YouCookII datasets, where we achieved 10.12 and 6.58 METEOR score, respectively.

Motivation & Objective

• Motivate dense video captioning that tightly couples event localization with description generation.
• Develop an end-to-end transformer model to enable language information to influence event proposals.
• Introduce a differentiable mask to align proposal regions with captioning attention for consistent training.

Proposed method

• Use a video encoder with self-attention to produce frame representations.
• Implement a proposal decoder with anchors to produce event proposals (start, end, confidence).
• Implement a captioning decoder with a masking network that converts proposals into a differentiable mask guiding attention to the current event.
• Apply a gated continuous/discrete masking function to fuse proposal predictions with the captioning process.
• Train end-to-end with a multi-task loss including regression, mask, proposal, and captioning losses.

Experimental results

Research questions

• RQ1 Can an end-to-end Transformer framework enable language information to directly influence event proposals in dense video captioning?
• RQ2 Does a differentiable masking mechanism improve consistency between proposed events and generated captions during training?
• RQ3 Do self-attention based encoders and decoders improve long-range dependency modeling for dense video captioning compared to RNN-based approaches?
• RQ4 How does the end-to-end model perform on ActivityNet Captions and YouCookII compared to LSTM-based baselines?

Key findings

• The End-to-end Masked Transformer outperforms LSTM-based baselines on ActivityNet Captions and YouCookII in captioning and dense-video metrics.
• On ActivityNet Captions with learned proposals, the model achieves BLEU-3 of 4.76, BLEU-4 of 2.23, and METEOR of 9.56.
• Compared to Bi-LSTM and TempoAttn baselines, the end-to-end model with masking yields higher METEOR and BLEU scores in validation.
• Using ground-truth proposals, increasing transformer depth yields incremental gains in captioning metrics, with 2-layer serving as a good balance.
• Self-attention based context encoding improves event proposal recall and overall dense captioning performance versus Bi-LSTM baselines.
• The differentiable masking scheme (and its gated variant) helps maintain consistency between proposed events and generated captions during training.