[Paper Review] Theoretical Foundations of t-SNE for Visualizing High-Dimensional Clustered Data

T. Tony Cai, Rong Ma|arXiv (Cornell University)|May 16, 2021

Advanced Clustering Algorithms Research30 references96 citations

TL;DR

The paper provides a rigorous, gradient-descent-based theoretical framework for t-SNE, linking early exaggeration to power iterations on a graph Laplacian and detailing embedding-stage dynamics and implicit regularization.

ABSTRACT

This paper investigates the theoretical foundations of the t-distributed stochastic neighbor embedding (t-SNE) algorithm, a popular nonlinear dimension reduction and data visualization method. A novel theoretical framework for the analysis of t-SNE based on the gradient descent approach is presented. For the early exaggeration stage of t-SNE, we show its asymptotic equivalence to power iterations based on the underlying graph Laplacian, characterize its limiting behavior, and uncover its deep connection to Laplacian spectral clustering, and fundamental principles including early stopping as implicit regularization. The results explain the intrinsic mechanism and the empirical benefits of such a computational strategy. For the embedding stage of t-SNE, we characterize the kinematics of the low-dimensional map throughout the iterations, and identify an amplification phase, featuring the intercluster repulsion and the expansive behavior of the low-dimensional map, and a stabilization phase. The general theory explains the fast convergence rate and the exceptional empirical performance of t-SNE for visualizing clustered data, brings forth interpretations of the t-SNE visualizations, and provides theoretical guidance for applying t-SNE and selecting its tuning parameters in various applications.

Motivation & Objective

• Motivate and justify the theoretical foundations of t-SNE for visualizing high-dimensional clustered data.
• Develop a gradient-descent-based framework that connects early exaggeration to spectral graph properties.
• Characterize embedding-stage dynamics to explain fast convergence and cluster separation.
• Provide practical guidance on initialization and tuning parameters for clustered data.

Proposed method

• Define P and Q similarity matrices and the KL-divergence objective of t-SNE.
• Reformulate the early exaggeration update as a linear operator involving the Laplacian of a graph built from S^{(k)}_{\alpha}.
• Show asymptotic equivalence between early exaggeration steps and power iterations of I_n - h L(S^{(k)}_{\alpha}).
• Establish localization, implicit regularization, and conditions for early stopping in the early exaggeration stage.
• Analyze embedding-stage kinematics to identify amplification and stabilization phases.
• Provide a framework to interpret t-SNE outputs and offer parameter-tuning guidance based on cluster structure.

Experimental results

Research questions

• RQ1 What is the theoretical mechanism by which early exaggeration enhances cluster visualization in t-SNE?
• RQ2 How does early exaggeration relate to power iterations and Laplacian spectral properties?
• RQ3 What are the embedding-stage dynamics that drive fast convergence and cluster separation?
• RQ4 How should initialization and tuning parameters be chosen for clustered data to avoid overshooting or artifacts?
• RQ5 What practical interpretations and guidelines arise from the theory for applying t-SNE in real data?

Key findings

• Early exaggeration is asymptotically equivalent to power iterations on the graph Laplacian related to P, with an implicit spectral clustering interpretation.
• For weakly clustered data, early stopping acts as implicit regularization to avoid overshooting.
• Embedding stage exhibits an amplification phase with intercluster repulsion and global expansion, followed by a stabilization phase.
• When data are strongly clustered, the method aligns with the Laplacian null space, yielding cluster-separated embeddings.
• The theory provides data-adaptive guidance for initialization and tuning parameters, and supports practical visualization interpretations.