[Paper Review] Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization

Yingfan Wang, Haiyang Huang+2 more|arXiv (Cornell University)|Dec 8, 2020

Data Visualization and Analytics46 references241 citations

TL;DR

The paper analyzes why local and global structure are preserved in DR methods, introduces principles for effective loss design, and presents PaCMAP as a new algorithm that preserves both structures. It also offers guidance on graph component choices and initialization effects.

ABSTRACT

Dimension reduction (DR) techniques such as t-SNE, UMAP, and TriMAP have demonstrated impressive visualization performance on many real world datasets. One tension that has always faced these methods is the trade-off between preservation of global structure and preservation of local structure: these methods can either handle one or the other, but not both. In this work, our main goal is to understand what aspects of DR methods are important for preserving both local and global structure: it is difficult to design a better method without a true understanding of the choices we make in our algorithms and their empirical impact on the lower-dimensional embeddings they produce. Towards the goal of local structure preservation, we provide several useful design principles for DR loss functions based on our new understanding of the mechanisms behind successful DR methods. Towards the goal of global structure preservation, our analysis illuminates that the choice of which components to preserve is important. We leverage these insights to design a new algorithm for DR, called Pairwise Controlled Manifold Approximation Projection (PaCMAP), which preserves both local and global structure. Our work provides several unexpected insights into what design choices both to make and avoid when constructing DR algorithms.

Motivation & Objective

• Identify which aspects of DR loss functions influence local vs. global structure preservation.
• Understand how graph components (neighbors, mid-near, further points) affect embeddings.
• Develop a simple loss that preserves both local and global structure.
• Show how initialization and scaling impact DR performance.
• Introduce PaCMAP and demonstrate its ability to preserve both structures.

Proposed method

• Review and compare t-SNE, UMAP, TriMap, and PaCMAP through loss-function and graph-component analysis.
• Propose a unified DR objective that decomposes loss into high- and low-dimensional components tied to graph structures.
• Introduce the rainbow figure as a visualization tool to compare loss mechanisms across methods.
• Derive principles for loss design that favor local structure preservation and global structure awareness.
• Define and implement PaCMAP with a three-stage weighting scheme for neighbor, mid-near, and further-point pairs.

Experimental results

Research questions

• RQ1 What elements of the DR loss functions are important for preserving local versus global structure?
• RQ2 Can a simple loss function preserve both local and global structure in a unified framework?
• RQ3 How do choices of graph components (neighbors, mid-near, further points) impact embedding quality?
• RQ4 What role does initialization play in DR performance for global structure preservation?
• RQ5 Can a new algorithm (PaCMAP) embody the identified principles and outperform existing methods in preserving both structures?

Key findings

• Loss function design and graph-component choices critically influence local and global structure preservation in DR embeddings.
• A unified DR objective can be decomposed into high- and low-dimensional terms across graph components, enabling principled analysis.
• PaCMAP’s loss, guided by the proposed principles, preserves both local and global structure and uses mid-near pairs to help maintain global relationships.
• Initialization can have a surprising impact on global-structure preservation across several DR methods.
• PaCMAP demonstrates competitive performance relative to UMAP for local structure and TriMap for global structure on representative datasets.