[Paper Review] TextTopicNet - Self-Supervised Learning of Visual Features Through Embedding Images on Semantic Text Spaces

Yash Patel, Lluís Gómez+4 more|arXiv (Cornell University)|Jul 4, 2018

Image Retrieval and Classification Techniques55 references4 citations

TL;DR

TextTopicNet proposes a self-supervised method to learn visual features by training a CNN to predict the semantic topic context—derived from LDA-embedded Wikipedia articles—in which an image is likely to appear as an illustration. By leveraging freely available text-image pairs from multilingual Wikipedia, the model achieves state-of-the-art performance in image classification, object detection, and cross-modal retrieval without any human-annotated labels.

ABSTRACT

The immense success of deep learning based methods in computer vision heavily relies on large scale training datasets. These richly annotated datasets help the network learn discriminative visual features. Collecting and annotating such datasets requires a tremendous amount of human effort and annotations are limited to popular set of classes. As an alternative, learning visual features by designing auxiliary tasks which make use of freely available self-supervision has become increasingly popular in the computer vision community. In this paper, we put forward an idea to take advantage of multi-modal context to provide self-supervision for the training of computer vision algorithms. We show that adequate visual features can be learned efficiently by training a CNN to predict the semantic textual context in which a particular image is more probable to appear as an illustration. More specifically we use popular text embedding techniques to provide the self-supervision for the training of deep CNN. Our experiments demonstrate state-of-the-art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or naturally-supervised approaches.

Motivation & Objective

• To address the high cost and limited scalability of human-annotated datasets in deep learning for computer vision.
• To explore whether unstructured, naturally co-occurring text from illustrated articles can serve as a free, scalable supervisory signal for visual representation learning.
• To investigate whether topic-level semantic representations (via LDA) are more effective than word-level or document-level embeddings for self-supervised visual feature learning.
• To demonstrate that self-supervised visual features can match or exceed those from supervised pre-training on ImageNet, without using any human-annotated labels.

Proposed method

• The method uses a CNN to predict the semantic topic vector of a Wikipedia article in which a given image appears as an illustration.
• Text embeddings are generated using Latent Dirichlet Allocation (LDA) to model the topic-level semantic context of full articles.
• The CNN is trained end-to-end using contrastive loss to align image features with the corresponding LDA-encoded text topic vectors.
• The model is pre-trained on a large-scale, multilingual Wikipedia corpus containing over 1 million image-text pairs from diverse domains.
• Features from the final layer (prob layer) are used for downstream tasks such as image classification, object detection, and cross-modal retrieval.
• No human-annotated labels are used during pre-training; supervision comes entirely from the co-occurrence of images and their textual context in Wikipedia articles.

Experimental results

Research questions

• RQ1 Can semantic text embeddings from unstructured, naturally occurring Wikipedia articles serve as an effective self-supervisory signal for visual representation learning?
• RQ2 Is topic-level semantic representation (via LDA) more effective than word-level or document-level text embeddings for self-supervised visual feature learning?
• RQ3 Can self-supervised visual features learned from Wikipedia articles generalize to downstream tasks such as image classification and object detection without fine-tuning?
• RQ4 How does the performance of TextTopicNet compare to supervised and unsupervised self-supervised baselines on standard benchmarks?

Key findings

• TextTopicNet achieves a mean average precision (mAP) of 38.94 on the Wikipedia dataset, outperforming all unsupervised methods and approaching the performance of supervised methods.
• On the ImageCLEF dataset, TextTopicNet achieves a mAP of 38.87, demonstrating strong generalization across different multilingual and multi-domain settings.
• The model achieves state-of-the-art performance in image classification and object detection on PASCAL VOC 2007, even without any fine-tuning on the VOC dataset.
• Qualitative results show that features from the topic space layer (prob layer) enable semantically relevant image retrieval, even for polysemic queries like 'airplane' and 'fighter'.
• The method successfully retrieves images semantically related to a query, such as 'airplane' and 'flight', despite no explicit supervision on semantic similarity.
• The model learns generic, broad-topic visual features that still support fine-grained recognition, as evidenced by correct class retrieval on PASCAL VOC 2007 without any class-specific supervision.