[Paper Review] Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples

Angie Boggust, Brandon Carter+1 more|arXiv (Cornell University)|Dec 10, 2019

Data Visualization and Analytics4 citations

TL;DR

The Embedding Comparator is an interactive visualization system that enables systematic comparison of two embedding spaces by computing and displaying similarity scores for local neighborhoods across models. It uses small multiples (dominoes) to highlight global structural differences and fine-grained neighborhood contrasts, significantly accelerating insight discovery in NLP, vision, and multimodal applications compared to manual inspection.

ABSTRACT

Embeddings mapping high-dimensional discrete input to lower-dimensional continuous vector spaces have been widely adopted in machine learning applications as a way to capture domain semantics. Interviewing 13 embedding users across disciplines, we find comparing embeddings is a key task for deployment or downstream analysis but unfolds in a tedious fashion that poorly supports systematic exploration. In response, we present the Embedding Comparator, an interactive system that presents a global comparison of embedding spaces alongside fine-grained inspection of local neighborhoods. It systematically surfaces points of comparison by computing the similarity of the $k$-nearest neighbors of every embedded object between a pair of spaces. Through case studies across multiple modalities, we demonstrate our system rapidly reveals insights, such as semantic changes following fine-tuning, language changes over time, and differences between seemingly similar models. In evaluations with 15 participants, we find our system accelerates comparisons by shifting from laborious manual specification to browsing and manipulating visualizations.

Motivation & Objective

• To address the lack of systematic, interactive tools for comparing embedding spaces in machine learning workflows.
• To support both global structural analysis and local neighborhood inspection in a unified interface.
• To reduce reliance on ad hoc, manual selection of objects for comparison, which hinders hypothesis testing and insight discovery.
• To enable rapid, systematic identification of semantic changes across models, such as those from fine-tuning or temporal shifts.
• To provide a generalizable, domain-agnostic metric for comparing embeddings without requiring alignment or task-specific definitions.

Proposed method

• Computes a local neighborhood similarity (LNS) score for each embedded object by measuring the Jaccard similarity of its k-nearest neighbors across two embedding spaces.
• Visualizes the LNS score distribution across all objects using a histogram and color-codes global projection plots (e.g., PCA) to highlight high- and low-similarity regions.
• Presents local neighborhoods as 'dominoes'—small multiples showing shared and unique neighbors in each model, with color-coding for common (green) and unique (purple) neighbors.
• Initial view defaults to lists of the most and least similar objects, based on LNS scores, to bootstrap exploration and reduce cognitive load.
• Supports multiple modalities, including text, images, and molecular structures, by using concise visual representations for embedded objects.
• Enables interactive filtering and exploration, allowing users to browse, select, and drill into specific objects without manual specification.

Experimental results

Research questions

• RQ1 How can embedding comparison be made more systematic and less reliant on ad hoc object selection?
• RQ2 What visualization techniques best support both global structure and local neighborhood comparison in embedding spaces?
• RQ3 Can a general-purpose similarity metric for local neighborhoods reveal meaningful semantic differences without alignment or task-specific tuning?
• RQ4 How does the Embedding Comparator improve the speed and depth of insight discovery compared to existing workflows?
• RQ5 To what extent can small multiples effectively scale to the exploration of large embedding spaces?

Key findings

• The Embedding Comparator accelerated comparison workflows by shifting from manual, ad hoc selection to interactive browsing and manipulation of visualizations, as validated in a 15-participant evaluation.
• The system revealed semantic changes after fine-tuning, such as shifts in word associations (e.g., 'artificial' gaining 'intelligence' and 'machines' as neighbors), with minimal user input.
• Local neighborhood dominoes effectively highlighted both shared and unique neighbors, enabling rapid detection of model-specific biases or semantic drift.
• The LNS metric successfully identified the most and least similar objects across models without requiring model alignment or domain-specific tuning.
• Case studies demonstrated the system’s utility across diverse modalities, including NLP, computer vision, and molecular embeddings, with consistent insight generation.
• Users reported improved confidence in detecting unexpected differences and a stronger sense of comprehensive understanding of embedding space variations.