[Paper Review] An Analysis of the t-SNE Algorithm for Data Visualization

Sanjeev Arora, Wei Hu+1 more|arXiv (Cornell University)|Mar 5, 2018

Topological and Geometric Data Analysis28 references82 citations

TL;DR

This paper provides a formal framework for data visualization via 2D embeddings and proves provable guarantees for t-SNE under well-separated, spherical cluster assumptions, including applications to mixtures of Gaussians and log-concave distributions.

ABSTRACT

A first line of attack in exploratory data analysis is data visualization, i.e., generating a 2-dimensional representation of data that makes clusters of similar points visually identifiable. Standard Johnson-Lindenstrauss dimensionality reduction does not produce data visualizations. The t-SNE heuristic of van der Maaten and Hinton, which is based on non-convex optimization, has become the de facto standard for visualization in a wide range of applications. This work gives a formal framework for the problem of data visualization - finding a 2-dimensional embedding of clusterable data that correctly separates individual clusters to make them visually identifiable. We then give a rigorous analysis of the performance of t-SNE under a natural, deterministic condition on the "ground-truth" clusters (similar to conditions assumed in earlier analyses of clustering) in the underlying data. These are the first provable guarantees on t-SNE for constructing good data visualizations. We show that our deterministic condition is satisfied by considerably general probabilistic generative models for clusterable data such as mixtures of well-separated log-concave distributions. Finally, we give theoretical evidence that t-SNE provably succeeds in partially recovering cluster structure even when the above deterministic condition is not met.

Motivation & Objective

• Formalize the visualization task as finding a 2D embedding that makes ground-truth clusters visually separable.
• Identify deterministic conditions under which t-SNE provably yields a full visualization of clusterable data.
• Show that these conditions are satisfied by mixtures of Gaussians and log-concave distributions.
• Provide evidence that t-SNE can partially recover cluster structure even when conditions are not fully met.

Proposed method

• Define precise notions of visible and full visualization for 2D embeddings of clusterable data.
• Introduce γ-spherical and γ-well-separated data conditions and relate them to cluster visualization guarantees.
• Analyze t-SNE with early exaggeration and derive update dynamics analogous to the power method on a pairwise similarity matrix.
• Prove that under the conditions, cluster centroids stay separated while intra-cluster points shrink, yielding a full visualization.
• Apply the theory to mixtures of isotropic Gaussians and log-concave distributions, deriving separation scales (e.g., ~d1/4 for Gaussians) that guarantee visualization.

Experimental results

Research questions

• RQ1 Under what deterministic conditions on the data does t-SNE with early exaggeration produce a full 2D visualization of clusters?
• RQ2 Do these conditions hold for common generative models like mixtures of Gaussians or log-concave distributions?
• RQ3 Can t-SNE visualize at least some cluster when the ideal conditions fail?
• RQ4 How does the visualization guarantee compare to linear dimensionality reduction methods in high dimensions?

Key findings

• For γ-spherical and γ-well-separated clusterable data with cluster size at least 0.1(n/k) and k ≪ n1/5, t-SNE with early exaggeration outputs a full visualization with high probability.
• The analysis shows intra-cluster points shrink and inter-cluster centroids remain separated, ensuring visual distinctness of clusters.
• Corollaries establish that i.i.d. samples from a mixture of k isotropic Gaussians with means separated by ~d1/4 yield a full visualization, as do mixtures of isotropic log-concave distributions with separation ~d5/12.
• t-SNE's behavior is closely approximated by a power-method-like update on the pairwise affinity matrix, underpinning the theoretical guarantees.
• Even when the deterministic separation conditions fail, t-SNE can provide a partial visualization, recovering at least one cluster under certain concentric Gaussian mixtures.
• The results extend to non-isotropic log-concave mixtures and imply broader applicability beyond isotropic settings.