[Paper Review] Efficient Algorithms for t-distributed Stochastic Neighborhood Embedding

George C. Linderman, Manas Rachh+3 more|arXiv (Cornell University)|Dec 25, 2017

Stochastic Gradient Optimization Techniques19 references186 citations

TL;DR

This paper introduces FIt-SNE, a FFT-accelerated interpolation-based t-SNE approach for fast, scalable embedding of large high-dimensional datasets, plus an out-of-core PCA method (oocPCA) and late exaggeration to improve cluster separation.

ABSTRACT

t-distributed Stochastic Neighborhood Embedding (t-SNE) is a method for dimensionality reduction and visualization that has become widely popular in recent years. Efficient implementations of t-SNE are available, but they scale poorly to datasets with hundreds of thousands to millions of high dimensional data-points. We present Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE), which dramatically accelerates the computation of t-SNE. The most time-consuming step of t-SNE is a convolution that we accelerate by interpolating onto an equispaced grid and subsequently using the fast Fourier transform to perform the convolution. We also optimize the computation of input similarities in high dimensions using multi-threaded approximate nearest neighbors. We further present a modification to t-SNE called "late exaggeration," which allows for easier identification of clusters in t-SNE embeddings. Finally, for datasets that cannot be loaded into the memory, we present out-of-core randomized principal component analysis (oocPCA), so that the top principal components of a dataset can be computed without ever fully loading the matrix, hence allowing for t-SNE of large datasets to be computed on resource-limited machines.

Motivation & Objective

• Improve the computational speed of t-SNE for very large datasets (hundreds of thousands to millions of points).
• Develop efficient methods for computing input similarities and repulsive forces in t-SNE.
• Enhance cluster separation in t-SNE embeddings via late exaggeration and related strategies.
• Provide out-of-core PCA to enable t-SNE on datasets that do not fit into memory.
• Offer practical visualization tools (1D/2D t-SNE heatmaps) and scalable software implementations.

Proposed method

• FFT-accelerated interpolation-based computation of the repulsive forces by interpolating onto an equispaced grid and using FFTs to perform the resulting convolutions.
• Piecewise polynomial kernel interpolation with low-order (p=3) interpolation to approximate N-body interactions.
• Use of translationally invariant kernels (K1 and K2) and Toeplitz structure to enable FFT acceleration.
• Approximate nearest neighbor strategies (ANNOY) to compute input similarities and reduce attractor computations.
• Introduction of late exaggeration (alpha>1) during later iterations to improve cluster separation.
• Out-of-core randomized PCA (oocPCA) to compute top components without loading the full data matrix into memory.

Experimental results

Research questions

• RQ1 Can t-SNE be scaled to datasets with millions of points while preserving embedding quality?
• RQ2 How can the repulsive force computation in t-SNE be accelerated beyond Barnes-Hut without sacrificing accuracy?
• RQ3 Can input similarity computation be made faster using approximate nearest neighbors without losing essential local geometry?
• RQ4 Does the use of late exaggeration improve cluster separability and interpretability of t-SNE embeddings?
• RQ5 Is there a memory-efficient pipeline (out-of-core PCA) enabling t-SNE on datasets that do not fit in memory?

Key findings

• FIt-SNE achieves up to 15x speedups in 1D and 30x in 2D compared to Barnes-Hut t-SNE on large datasets (e.g., 1 million points).
• FFT-accelerated interpolation reduces the repulsive force computation from quadratic to near-linear time with a complexity of O(N·p + (N_int·p) log(N_int·p)).
• Using approximate nearest neighbors (ANNOY) for input similarities speeds up attraction computations while preserving local structure.
• Late exaggeration (alpha > 1) in later iterations improves cluster separability and interpretability of t-SNE embeddings.
• t-SNE heatmaps enable 1D embeddings to convey gene expression information in a compact, heatmap-like visualization.
• An out-of-core PCA (oocPCA) implementation computes top components without loading the full matrix, demonstrated on very large matrices with strong memory constraints.