[Paper Review] I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision

Sophia Gu, Christopher M. Clark+1 more|arXiv (Cornell University)|Nov 17, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes CLOSE, a method to train visual task models using only text supervision by leveraging the shared semantic embedding space of contrastive vision-language models. By freezing vision and language encoders and training solely on text data with adaptive noise, the approach achieves near-state-of-the-art performance on image captioning, visual entailment, VQA, and visual news captioning—surpassing prior text-only methods by over 9 points on captioning and entailment, and by over 30 points on visual news captioning.

ABSTRACT

Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether it is possible to learn those skills from text data and then transfer them to vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study strategies to mitigate this concern. We produce models using only text training data on four representative tasks: image captioning, visual entailment, visual question answering and visual news captioning, and evaluate them on standard benchmarks using images. We find these models perform close to models trained on images, while surpassing prior work for captioning and visual entailment in this text-only setting by over 9 points, and outperforming all prior work on visual news by over 30 points. We also showcase a variety of stylistic image captioning models that are trained using no image data and no human-curated language data, but instead using readily-available text data from books, the web, or language models.

Motivation & Objective

• To investigate whether high-level visual understanding skills can be learned from text data alone and transferred to vision tasks without any visual training data.
• To address the domain shift between image and text embeddings in contrastive models, which can hinder cross-modal transfer performance.
• To develop a method that enables zero-shot cross-modal transfer by leveraging shared semantic spaces from contrastive pre-training.
• To demonstrate the feasibility of training stylistic image captioning models using only text data, including data from books, the web, or language models.
• To establish a scalable, low-cost alternative to expensive visual data collection by using readily available text data for training visual models.

Proposed method

• Leverage a contrastively pre-trained vision-and-language (V&L) model to create a shared semantic embedding space for images and text.
• Freeze the vision and language encoders during training and use only text inputs during the training phase to learn task-specific representations.
• Apply a text-to-image transfer strategy by substituting the text embedding with an image embedding at inference time, enabling zero-shot cross-modal generalization.
• Introduce adaptive noise injection (e.g., Gaussian noise) to the text embeddings during training to mitigate modality-specific distribution shifts and improve robustness.
• Use adapters to fine-tune the model’s head while keeping the frozen encoders, enabling task-specific adaptation without retraining the backbone.
• Train models on diverse text data sources—including books, web text, and LLM-generated text—without requiring any paired image-caption annotations.

Figure 1 : Overview of CLOSE . During training, input text is encoded into a vector with a text encoder and adapted with an adaptation method. A model learns to use the vector to perform a task such as VQA, captioning, or visual entailment. During testing, an input image is encoded with an image enc

Experimental results

Research questions

• RQ1 Can high-level visual understanding skills such as captioning, visual question answering, and entailment be learned from text data alone and transferred to vision tasks without any visual supervision?
• RQ2 How do systematic differences in the embedding spaces of images and text in contrastive models affect cross-modal transfer performance, and how can these be mitigated?
• RQ3 To what extent can models trained on text-only data achieve performance comparable to those trained with visual data on standard vision-and-language benchmarks?
• RQ4 Can stylistic image captioning models be effectively trained using only text data, including data from language models or diverse textual sources?
• RQ5 Does the use of adaptive noise or other domain-invariant regularization techniques significantly improve zero-shot cross-modal transfer in the absence of visual data?

Key findings

• CLOSE models trained exclusively on text data achieve performance within 5-10 points of fully supervised models on image captioning, visual entailment, and VQA, demonstrating strong zero-shot transfer capability.
• The method surpasses prior text-only approaches by 17 CIDEr points on image captioning (95.4 vs. 78.2) and by 9 points on visual entailment (75.9 vs. 66.6), setting new SOTA in the text-only setting.
• On the visual news captioning benchmark, the text-only model outperforms the previous best result—trained with visual data—by 30.3 CIDEr points (80.8 vs. 50.5), highlighting the method’s effectiveness.
• Models trained on text data generated by large language models (e.g., GPT-3) achieve strong performance, requiring only prompt engineering and no human-annotated image data.
• The addition of Gaussian noise during training significantly improves robustness to modality distribution shifts, outperforming baseline methods without such regularization.
• Stylistic captioning models trained on diverse text sources (e.g., books, reviews, LLM outputs) generate accurate and stylistically consistent captions without any paired image data.

Figure 2 : Using CLOSE to learn stylistic captioning without image data. Text examples of the desired style are gathered from sources such as the web, books, or GPT-3 . Models are trained on text only and then applied to images.