[Paper Review] Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning

Jia Cheng Hu, Roberto Cavicchioli+1 more|arXiv (Cornell University)|Aug 13, 2022

Multimodal Machine Learning ApplicationsComputer Science55 references20 citations

TL;DR

The paper introduces the Expansion mechanism to process inputs with arbitrary sequence lengths, enabling ExpansionNet v2 to surpass baselines on MS COCO 2014 in offline and online evaluations while achieving faster end-to-end training.

ABSTRACT

We introduce a method called the Expansion mechanism that processes the input unconstrained by the number of elements in the sequence. By doing so, the model can learn more effectively compared to traditional attention-based approaches. To support this claim, we design a novel architecture ExpansionNet v2 that achieved strong results on the MS COCO 2014 Image Captioning challenge and the State of the Art in its respective category, with a score of 143.7 CIDErD in the offline test split, 140.8 CIDErD in the online evaluation server and 72.9 AllCIDEr on the nocaps validation set. Additionally, we introduce an End to End training algorithm up to 2.8 times faster than established alternatives. Source code available at: https://github.com/jchenghu/ExpansionNet_v2

Motivation & Objective

• Motivation: fixed input sequence length may bottleneck stateless architectures in image captioning.
• Propose Expansion mechanism to distribute input content over variable-length sequences during forward and recover original length backward.
• Develop ExpansionNet v2 architecture that leverages Static and Dynamic Expansion without heavy reliance on traditional attention.

Proposed method

• Introduce Expansion mechanism that expands input into an arbitrary number of elements in forward pass and retracts to original length in backward pass.
• Define Static Expansion and Dynamic Expansion (auto-regressive and bidirectional variants) with Forward and Backward Expansion steps.
• Use BroadSum operator to compute expansion queries and biases in Dynamic Expansion.
• Integrate Expansion layers into an encoder–decoder on top of Swin-Transformer backbone (ExpansionNet v2).
• Train with two-stage objectives: Cross-Entropy loss followed by CIDEr-D optimized reinforcement learning (SCST).
• Adopt a four-step End-to-End training schedule with backbone freezing and subsequent fine-tuning.

Experimental results

Research questions

• RQ1 Does allowing variable or expanded sequence lengths in processing improve image captioning performance compared to fixed-length attention-based models?
• RQ2 Can ExpansionNet v2 achieve competitive or state-of-the-art results on MS COCO 2014 in offline and online evaluations without relying on heavy pre-training?
• RQ3 What is the computational trade-off of Expansion mechanisms in terms of training speed and FLOPs?
• RQ4 How do Static and Dynamic Expansion contribute individually and jointly to performance?

Key findings

• ExpansionNet v2 achieves 143.7 CIDEr-D offline and 140.8 CIDEr-D online on MS COCO 2014.
• ExpansionNet v2 attains 72.9 AllCIDEr on nocaps validation set.
• Static and Dynamic Expansion outperform the baseline across metrics, with Dynamic Expansion providing particularly notable gains.
• Online and offline results show ExpansionNet v2 surpasses several single-model baselines on Karpathy split (e.g., CIDEr-D improvements).
• The model is faster to train end-to-end than many alternatives, with training times notably shorter than several non-generative models and orders of magnitude faster than many generative ones.
• ExpansionNet v2 remains competitive with state-of-the-art V+L pre-training models while using far less data and smaller model size.