[Paper Review] Visual Conceptual Blending with Large-scale Language and Vision Models

Songwei Ge, Devi Parikh|arXiv (Cornell University)|Jun 27, 2021

Multimodal Machine Learning Applications32 references4 citations

TL;DR

This paper proposes a framework that leverages large-scale language and vision models to generate creative visual conceptual blends by first using prompt-engineered language models to reason about relevant objects and blending properties, then generating images via text-to-image diffusion models. Results show that large language models outperform knowledge bases in conceptual association, and diffusion-based image generators significantly surpass GANs in visual quality and fidelity of blends.

ABSTRACT

We ask the question: to what extent can recent large-scale language and image generation models blend visual concepts? Given an arbitrary object, we identify a relevant object and generate a single-sentence description of the blend of the two using a language model. We then generate a visual depiction of the blend using a text-based image generation model. Quantitative and qualitative evaluations demonstrate the superiority of language models over classical methods for conceptual blending, and of recent large-scale image generation models over prior models for the visual depiction.

Motivation & Objective

• To investigate whether large-scale language and vision models can effectively perform visual conceptual blending—combining two distinct objects into a novel, coherent conceptual blend.
• To evaluate the superiority of modern large language models over classical knowledge bases in identifying relevant conceptual properties for blending.
• To compare the visual quality and conceptual fidelity of images generated by recent text-to-image diffusion models versus older GAN-based approaches.
• To develop a pipeline that integrates language models for reasoning and diffusion models for generation to produce high-quality, creative visual blends.

Proposed method

• The framework decomposes visual conceptual blending into two phases: reasoning and generation.
• In the reasoning phase, a prompt-engineered language model (e.g., BERT, GPT) is used to predict a relevant object and a blending property (e.g., 'sliced') for a given input object (e.g., 'moon').
• The prompt format is a masked language modeling task, such as 'a low budget film is [MASK]', to elicit relevant properties.
• In the generation phase, the resulting textual description (e.g., 'the moon sliced like an orange') is used as input to text-to-image diffusion models (e.g., BigSleep, DeepDaze, DF-GAN) to generate visual depictions.
• Human evaluation on Amazon Mechanical Turk is used to compare image quality, recognition of input objects, and conceptual blending fidelity across models.
• A simile dataset with 66,442 property-object pairs is constructed and used to quantitatively evaluate language model performance against ConceptNet.

Experimental results

Research questions

• RQ1 Can large-scale language models outperform traditional knowledge bases in identifying relevant conceptual properties for blending two objects?
• RQ2 To what extent do recent text-to-image diffusion models generate more coherent and aesthetically pleasing visual blends than older GAN-based models?
• RQ3 How well do the generated images preserve the identity of the original input objects while integrating the blending concept?
• RQ4 Can the combination of large language models and diffusion models produce visually and semantically meaningful conceptual blends that align with human perception of creativity?
• RQ5 What role do prompt engineering and model scale play in improving the quality of conceptual blending?

Key findings

• Large language models such as GPT achieve a P@1000 precision of 66.38% in predicting relevant properties for blending, significantly outperforming ConceptNet’s 5.90%.
• Among language models, GPT shows the highest performance, followed by BERT-base and BERT-large, indicating that model scale enhances conceptual reasoning.
• CLIP-based diffusion models (BigSleep and DeepDaze) are significantly preferred over DF-GAN in all human evaluation metrics, including object recognition, blending quality, and aesthetic appeal.
• BigSleep is preferred over DeepDaze, suggesting that the prior learned by BigGAN leads to better image generation for conceptual blends than SIREN-based models.
• The human evaluation results show statistically significant preferences (p < 0.05) for diffusion models on all six questions, confirming their superiority in visual conceptual blending.
• The framework successfully generates novel, coherent, and creative visual blends such as 'a tree made of blue and red blood vessels' and 'the moon sliced like an orange', demonstrating practical creative potential.