[Paper Review] Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions

Quanzeng You, Hailin Jin+1 more|arXiv (Cornell University)|Jan 30, 2018

Multimodal Machine Learning Applications31 references47 citations

TL;DR

The paper introduces two end-to-end models to inject sentiment into image captions, enabling controllable positive/negative captions without sacrificing visual-semantic alignment, and demonstrates superior performance over prior sentiment-captioning methods.

ABSTRACT

Automatic image captioning has recently approached human-level performance due to the latest advances in computer vision and natural language understanding. However, most of the current models can only generate plain factual descriptions about the content of a given image. However, for human beings, image caption writing is quite flexible and diverse, where additional language dimensions, such as emotion, humor and language styles, are often incorporated to produce diverse, emotional, or appealing captions. In particular, we are interested in generating sentiment-conveying image descriptions, which has received little attention. The main challenge is how to effectively inject sentiments into the generated captions without altering the semantic matching between the visual content and the generated descriptions. In this work, we propose two different models, which employ different schemes for injecting sentiments into image captions. Compared with the few existing approaches, the proposed models are much simpler and yet more effective. The experimental results show that our model outperform the state-of-the-art models in generating sentimental (i.e., sentiment-bearing) image captions. In addition, we can also easily manipulate the model by assigning different sentiments to the testing image to generate captions with the corresponding sentiments.

Motivation & Objective

• Motivate the need for sentiment-aware image captions beyond factual descriptions.
• Propose end-to-end models that inject sentiment into caption generation without degrading image-text alignment.
• Enable controllable sentiment generation by conditioning captions on explicit sentiment labels.
• Show that sentiment-aware models outperform state-of-the-art baselines on sentiment captioning tasks.

Proposed method

• Direct Injection: concatenate a sentiment unit (-1,0,1) to the RNN input at every generation step to bias word choice.
• Sentiment Flow: introduce a sentiment cell that propagates an initial sentiment signal through the LSTM, with sentiment loss guiding the final sentiment state.
• Train end-to-end using MS-COCO plus SentiCap data, with sentiment labels and optional sentiment loss.
• Use ResNet-152 as CNN encoder and a 256-d embedding with a 512-d RNN, trained with Adam optimizer.

Experimental results

Research questions

• RQ1 Can sentiment be injected into image captions while maintaining semantic correspondence with the image?
• RQ2 Which architectural scheme—direct injection or sentiment flow—better supports controllable sentiment captioning?
• RQ3 Does incorporating sentiment loss improve the model’s ability to distinguish and propagate sentiment across the caption sequence?
• RQ4 How well can the models generate captions that match a given sentiment label across positive and negative examples?

Key findings

• Both proposed models outperform cited baselines on sentiment-captioning benchmarks across standard metrics.
• Direct Injection yields stronger sentiment signal per-step and higher sentiment-captioning percentages, especially for negative captions.
• Sentiment Flow provides balanced performance across POS and NEG sets and benefits from sentiment loss in several configurations.
• The models support controllable generation by flipping the sentiment label at test time, yielding captions with matching sentiment words distributed across image content.