[Paper Review] SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text

A. P. Mathews, Lexing Xie+1 more|arXiv (Cornell University)|May 18, 2018

Multimodal Machine Learning ApplicationsComputer Science48 references19 citations

TL;DR

SemStyle proposes a novel framework for generating visually grounded, stylized image captions using unaligned text by separating semantic content from linguistic style. It uses a frame-semantic term representation and a unified language generator to produce descriptive, style-shifted captions, achieving 58.8% style distinctiveness and 41.9% story-like captions in human evaluation, with strong semantic retention (SPICE 0.144).

ABSTRACT

Linguistic style is an essential part of written communication, with the power to affect both clarity and attractiveness. With recent advances in vision and language, we can start to tackle the problem of generating image captions that are both visually grounded and appropriately styled. Existing approaches either require styled training captions aligned to images or generate captions with low relevance. We develop a model that learns to generate visually relevant styled captions from a large corpus of styled text without aligned images. The core idea of this model, called SemStyle, is to separate semantics and style. One key component is a novel and concise semantic term representation generated using natural language processing techniques and frame semantics. In addition, we develop a unified language model that decodes sentences with diverse word choices and syntax for different styles. Evaluations, both automatic and manual, show captions from SemStyle preserve image semantics, are descriptive, and are style shifted. More broadly, this work provides possibilities to learn richer image descriptions from the plethora of linguistic data available on the web.

Motivation & Objective

• To generate visually relevant, stylized image captions without requiring paired image-caption datasets.
• To disentangle semantic content from linguistic style in image captioning for greater flexibility and control.
• To leverage large-scale unaligned styled text (e.g., romance novels) to learn diverse writing styles without image alignment.
• To maintain high semantic relevance while enabling stylistic variation such as first-person, formal, or narrative tone.
• To develop a unified training strategy that combines descriptive and styled corpora for improved caption quality and style fidelity.

Proposed method

• Introduces a novel semantic term representation using lemmatized words with part-of-speech (POS) tags and verbs generalized via FrameNet to disentangle content from style.
• Employs a neural network-based term generator trained on image-caption pairs to extract a concise set of semantic terms from an image.
• Uses a unified language generator conditioned on target style to decode semantic terms into fluent, stylistically diverse sentences.
• Trains the language generator on both descriptive (MSCOCO) and styled (romance novel) corpora to learn style transfer without paired images.
• Applies joint training on multiple datasets to ensure semantic terms from all sources are accessible during inference, improving relevance.
• Uses a style classifier (CLF) and automated metrics (LM, GRULM) to evaluate style fidelity and content relevance during training and evaluation.

Experimental results

Research questions

• RQ1 Can a model generate stylistically diverse, visually grounded captions using only unaligned styled text and image data?
• RQ2 How can semantic content and linguistic style be effectively disentangled in image captioning to enable flexible style transfer?
• RQ3 To what extent can a language generator learn to produce narrative, first-person, or formal styles without paired image-caption annotations?
• RQ4 Does separating semantic term generation from style decoding improve both semantic relevance and stylistic diversity compared to end-to-end models?
• RQ5 Can joint training on descriptive and styled corpora improve caption relevance while preserving style fidelity?

Key findings

• SemStyle achieved 58.8% style distinctiveness in human evaluation, indicating strong stylistic control across diverse styles.
• The model produced 41.9% of captions judged as story-like, significantly outperforming baseline models like CNN+RNN-coco (6.2%) and neural-storyteller (52.6% but with 44.2% unrelated captions).
• SemStyle maintained high semantic relevance with a SPICE score of 0.144, comparable to the descriptive baseline (SPICE 0.144) and significantly better than TermRetrieval (SPICE 0.134) and neural-storyteller (SPICE 0.120).
• The model’s performance dropped when semantic terms were randomly ordered (SPICE 0.134) or when FrameNet/POS/lemmatization were removed, proving their role in disentangling style from content.
• Training on both MSCOCO and romance novel corpora improved relevance (SPICE 0.144) compared to training only on romance novels (SPICE 0.138), demonstrating the benefit of joint data utilization.
• The Kendall’s τ correlation between automated style metrics (CLF, LM, GRULM) and human judgments was 0.434 for CLF, indicating moderate to strong alignment with human perception of style.