[Paper Review] A Visual Embedding for the Unsupervised Extraction of Abstract Semantics

Dario García-Gasulla, Béjar, J.+5 more|arXiv (Cornell University)|Jul 31, 2015

Multimodal Machine Learning Applications18 references3 citations

TL;DR

This paper proposes a method to extract abstract visual semantics from deep neural network features by creating high-dimensional, sparse vector embeddings of image classes using GoogLeNet. It demonstrates that these embeddings capture semantic relationships—clustering semantically similar concepts (e.g., 118 dog breeds) and unsupervisedly separating high-level categories like living vs. non-living things—without linguistic supervision, enabling potential use in visual reasoning and cognitive systems.

ABSTRACT

Vector-space word representations obtained from neural network models have been shown to enable semantic operations based on vector arithmetic. In this paper, we explore the existence of similar information on vector representations of images. For that purpose we define a methodology to obtain large, sparse vector representations of image classes, and generate vectors through the state-of-the-art deep learning architecture GoogLeNet for 20K images obtained from ImageNet. We first evaluate the resultant vector-space semantics through its correlation with WordNet distances, and find vector distances to be strongly correlated with linguistic semantics. We then explore the location of images within the vector space, finding elements close in WordNet to be clustered together, regardless of significant visual variances (e.g. 118 dog types). More surprisingly, we find that the space unsupervisedly separates complex classes without prior knowledge (e.g. living things). Afterwards, we consider vector arithmetics. Although we are unable to obtain meaningful results on this regard, we discuss the various problem we encountered, and how we consider to solve them. Finally, we discuss the impact of our research for cognitive systems, focusing on the role of the architecture being used.

Motivation & Objective

• To explore whether deep neural network features can encode abstract visual semantics analogous to word vectors in NLP.
• To develop a methodology for generating high-dimensional, sparse vector representations of image classes from pre-trained CNN features.
• To evaluate whether these visual embeddings reflect linguistic and conceptual semantics without explicit supervision.
• To investigate the potential of these embeddings for higher-level cognitive tasks such as concept discovery and visual reasoning.
• To identify limitations in applying vector arithmetic to visual embeddings and propose future directions for improvement.

Proposed method

• Utilizes a pre-trained GoogLeNet (22-layer CNN) to extract activation values from all layers for each image in a dataset.
• Aggregates and normalizes feature activations across all layers to form a single, high-dimensional vector per image class.
• Constructs a sparse, over-one-million-dimensional vector space representation for 20,000 ImageNet image classes.
• Employs clustering algorithms to analyze semantic structure in the vector space, assessing similarity based on semantic distance.
• Evaluates semantic coherence by correlating vector distances with WordNet semantic distances.
• Analyzes the spatial organization of image classes in the embedding space to detect unsupervised clustering of abstract concepts.

Experimental results

Research questions

• RQ1 Can deep CNN features encode abstract visual semantics beyond simple classification?
• RQ2 To what extent do vector distances in the embedding space correlate with human-annotated semantic distances (e.g., WordNet)?
• RQ3 Can the embedding space unsupervisedly cluster semantically related image classes despite visual variation (e.g., 118 dog breeds)?
• RQ4 Does the embedding space naturally separate high-level semantic categories such as living vs. non-living things without prior training on such labels?
• RQ5 What are the limitations of applying vector arithmetic (e.g., analogies) to visual embeddings, and how can they be addressed?

Key findings

• Vector distances in the learned embedding space show strong correlation with WordNet semantic distances, indicating that the embedding preserves linguistic semantics.
• Semantically similar image classes—such as all 118 dog breeds—cluster together in the embedding space despite significant visual variation in size, color, and pose.
• The model unsupervisedly separates high-level categories such as living organisms from non-living entities, suggesting the capture of abstract, non-visual features.
• The embedding space demonstrates the ability to represent abstract semantics by aggregating sub-symbolic visual features into symbolic-like conceptual representations.
• Vector arithmetic operations on the visual embeddings yielded no meaningful results, highlighting challenges in applying such operations to visual data.
• The methodology successfully translates sub-symbolic pixel-level data into high-level symbolic knowledge, enabling potential applications in visual reasoning and zero-shot learning.