[Paper Review] Efficient CNN-LSTM based Image Captioning using Neural Network Compression

Harshit Rampal, Aman Mohanty|arXiv (Cornell University)|Dec 17, 2020

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes an end-to-end compression pipeline for CNN-LSTM image captioning models using magnitude-based pruning and quantization, achieving a 73.1% reduction in model size, 71.3% faster inference, and a 7.7% BLEU score improvement over the uncompressed baseline when using VGG16 and LSTM with quantization-aware training on the Flickr8k dataset.

ABSTRACT

Modern Neural Networks are eminent in achieving state of the art performance on tasks under Computer Vision, Natural Language Processing and related verticals. However, they are notorious for their voracious memory and compute appetite which further obstructs their deployment on resource limited edge devices. In order to achieve edge deployment, researchers have developed pruning and quantization algorithms to compress such networks without compromising their efficacy. Such compression algorithms are broadly experimented on standalone CNN and RNN architectures while in this work, we present an unconventional end to end compression pipeline of a CNN-LSTM based Image Captioning model. The model is trained using VGG16 or ResNet50 as an encoder and an LSTM decoder on the flickr8k dataset. We then examine the effects of different compression architectures on the model and design a compression architecture that achieves a 73.1% reduction in model size, 71.3% reduction in inference time and a 7.7% increase in BLEU score as compared to its uncompressed counterpart.

Motivation & Objective

• To enable real-time deployment of image captioning models on resource-constrained edge devices such as mobile and wearable systems.
• To investigate the effects of combining pruning and quantization on end-to-end CNN-LSTM architectures, which are typically large and computationally heavy.
• To optimize compression strategies for both encoder (VGG16/ResNet50) and decoder (LSTM) components to maintain or improve performance.
• To evaluate whether compressed models can outperform their full-precision counterparts in terms of BLEU score, model size, and inference speed.

Proposed method

• Applied magnitude-based pruning to both the CNN encoder (VGG16 or ResNet50) and LSTM decoder to remove less important weights.
• Implemented post-training quantization on the encoder and quantization-aware training on the decoder to reduce precision from 32-bit floating point to lower bit-widths.
• Designed and evaluated eight distinct compression architectures combining pruning and quantization across encoder and decoder components.
• Trained and fine-tuned models on the Flickr8k dataset with 5 captions per image, using BLEU score as the primary evaluation metric.
• Used a hybrid compression strategy that applied quantization-aware training to the decoder and post-training quantization to the encoder for optimal performance.
• Measured model size in MB, inference time per 2000 samples, and BLEU-1 scores to compare compressed vs. baseline models.

Experimental results

Research questions

• RQ1 Can end-to-end compression of a CNN-LSTM image captioning model using pruning and quantization maintain or improve performance on the Flickr8k dataset?
• RQ2 How does the combination of pruning and quantization affect model size, inference speed, and BLEU score in a joint encoder-decoder architecture?
• RQ3 Does quantization-aware training on the LSTM decoder yield better results than post-training quantization or pruning alone?
• RQ4 Is there a compression configuration that outperforms the full-precision baseline in both efficiency and accuracy?

Key findings

• The QVGG-QLSTM model, which uses quantized VGG16 encoder and quantized LSTM decoder, achieved a 7.7% increase in BLEU score compared to the baseline.
• This model reduced model size by 73.1% (from 578.4 MB to 155.39 MB) when using VGG16 as the encoder.
• Inference time was reduced by 71.3% (from 5.68 minutes to 1.63 minutes per 2000 samples) with the QVGG-QLSTM configuration.
• The quantized decoder with quantization-aware training outperformed both baseline and pruned variants, indicating that quantization alone can enhance performance.
• Pruning the encoder or decoder alone led to significant performance drops, with BLEU scores decreasing by up to 20.6% in some configurations.
• The best-performing model, QVGG-QLSTM, generated more accurate and contextually relevant captions than the baseline, as shown in qualitative comparisons on real images.