[Paper Review] Image Captioning using Deep Stacked LSTMs, Contextual Word Embeddings and Data Augmentation

Sulabh Katiyar, Samir Kumar Borgohain|arXiv (Cornell University)|Feb 22, 2021

Multimodal Machine Learning Applications9 citations

TL;DR

This paper proposes an image captioning model using Inception-ResNet-v2 for feature extraction, deep stacked LSTMs with contextual Komninos word embeddings, and data augmentation via flipping and perspective transformations. The approach achieves competitive performance without explicit attention mechanisms, outperforming baseline models on standard metrics due to improved feature representation and hierarchical word modeling.

ABSTRACT

Image Captioning, or the automatic generation of descriptions for images, is one of the core problems in Computer Vision and has seen considerable progress using Deep Learning Techniques. We propose to use Inception-ResNet Convolutional Neural Network as encoder to extract features from images, Hierarchical Context based Word Embeddings for word representations and a Deep Stacked Long Short Term Memory network as decoder, in addition to using Image Data Augmentation to avoid over-fitting. For data Augmentation, we use Horizontal and Vertical Flipping in addition to Perspective Transformations on the images. We evaluate our proposed methods with two image captioning frameworks- Encoder-Decoder and Soft Attention. Evaluation on widely used metrics have shown that our approach leads to considerable improvement in model performance.

Motivation & Objective

• To improve image captioning performance by leveraging deeper LSTM architectures and contextual word representations.
• To reduce overfitting in image captioning models through effective data augmentation techniques.
• To explore the effectiveness of intermediate convolutional features over global image embeddings for better spatial understanding.
• To evaluate whether contextual word embeddings and deep stacked LSTMs can compensate for the absence of attention mechanisms.
• To fine-tune both the CNN encoder and word embedding layers for improved representation learning.

Proposed method

• Utilizes Inception-ResNet-v2 as the CNN encoder to extract multi-scale feature maps from images, preserving spatial and regional information.
• Employs a three-layer deep stacked LSTM decoder to model long-range dependencies and complex linguistic structures in generated captions.
• Integrates Komninos word embeddings, which incorporate word co-occurrence and dependency context features, to generate contextually rich word representations.
• Applies data augmentation via horizontal/vertical flipping and perspective transformations to increase training data diversity and reduce overfitting.
• Fine-tunes both the CNN feature extractor and word embedding layer parameters during training to adapt representations to the captioning task.
• Evaluates the model under two frameworks: standard encoder-decoder and soft attention-based decoding, using cross-entropy loss to maximize conditional probability of ground-truth captions.

Experimental results

Research questions

• RQ1 Can a deep stacked LSTM decoder with contextual word embeddings achieve competitive performance without an explicit attention mechanism?
• RQ2 How does using intermediate convolutional features from Inception-ResNet-v2 compare to global image embeddings in captioning tasks?
• RQ3 To what extent does data augmentation with geometric transformations improve generalization and reduce overfitting in image captioning models?
• RQ4 Does fine-tuning the CNN encoder and word embedding layer lead to measurable performance gains?
• RQ5 How do contextual word embeddings enhance the model's ability to generate semantically accurate and contextually coherent captions?

Key findings

• The proposed encoder-decoder model without explicit attention outperforms or matches models using soft attention mechanisms on standard evaluation metrics.
• The use of Inception-ResNet-v2 for feature extraction leads to significant performance improvements over simpler CNNs due to richer feature representation.
• Deep stacked LSTMs (three layers) enable better modeling of complex linguistic dependencies compared to single-layer LSTMs.
• Contextual Komninos word embeddings improve word representation quality, enhancing semantic accuracy in generated captions.
• Data augmentation through flipping and perspective transformations effectively reduces overfitting and improves generalization on validation sets.
• Qualitative analysis shows the model generates more descriptive captions, correctly identifying object colors, actions, and scene details, though occasional word repetition occurs.