[論文レビュー] Attraction-Repulsion Spectrum in Neighbor Embeddings

Jan Niklas Böhm, Philipp Berens+1 more|arXiv (Cornell University)|Jul 17, 2020

Single-cell and spatial transcriptomics被引用数 32

ひとこと要約

論文はt-SNEでアトラクション強度（exaggeration parameter）を変化させることで隣接埋め込みの引力-反発スペクトラムを明らかにし、UMAPとForceAtlas2がこのスペクトラムにマップすることを示し、UMAPの挙動を負サンプリングで説明する。

ABSTRACT

Neighbor embeddings are a family of methods for visualizing complex high-dimensional datasets using $k$NN graphs. To find the low-dimensional embedding, these algorithms combine an attractive force between neighboring pairs of points with a repulsive force between all points. One of the most popular examples of such algorithms is t-SNE. Here we empirically show that changing the balance between the attractive and the repulsive forces in t-SNE using the exaggeration parameter yields a spectrum of embeddings, which is characterized by a simple trade-off: stronger attraction can better represent continuous manifold structures, while stronger repulsion can better represent discrete cluster structures and yields higher $k$NN recall. We find that UMAP embeddings correspond to t-SNE with increased attraction; mathematical analysis shows that this is because the negative sampling optimisation strategy employed by UMAP strongly lowers the effective repulsion. Likewise, ForceAtlas2, commonly used for visualizing developmental single-cell transcriptomic data, yields embeddings corresponding to t-SNE with the attraction increased even more. At the extreme of this spectrum lie Laplacian Eigenmaps. Our results demonstrate that many prominent neighbor embedding algorithms can be placed onto the attraction-repulsion spectrum, and highlight the inherent trade-offs between them.

研究の動機と目的

• Unify prominent neighbor-embedding methods (t-SNE, UMAP, ForceAtlas2, Laplacian eigenmaps) under a common attraction-repulsion framework.
• Investigate how varying attraction (exaggeration) alters the representation of continuous manifolds versus discrete clusters.
• Characterize where UMAP and FA2 lie on the attraction-repulsion spectrum and explain deviations via optimization choices.
• Quantify the trade-offs between local neighborhood preservation and global structure across methods and datasets.

提案手法

• Derive and compare gradient/loss formulations for t-SNE, UMAP, FA2, and Laplacian eigenmaps in a common NE framework.
• Introduce and analyze the exaggeration parameter rho that scales attractive forces in t-SNE. 3.3-3.5 sections provide the functional forms and connections to LE.
• Demonstrate, via MNIST and synthetic/developmental single-cell datasets, how increasing rho strengthens attraction and yields more continuum-like structure, while decreasing rho preserves clusters.
• Show that UMAP’s negative sampling effectively increases attraction (through reduced effective repulsion) and relate this to its embeddings on the spectrum.
• Use distance correlation and k-NN recall to quantify layout similarity and local neighborhood preservation across methods and rho values.
• Provide implementations and reproducible analysis using open-source code (ne-spectrum) and standard datasets.

Figure 1: Attraction-repulsion spectrum for the MNIST data. Different embeddings of the MNIST data set of hand-written digits ( $n=70\,000$ ); colors denote digits as shown in the t-SNE panel. Multiplying all attractive forces by an exaggeration factor $\rho$ yields a spectrum of embeddings. Values

実験結果

リサーチクエスチョン

• RQ1 How do different NE algorithms (t-SNE, UMAP, FA2, LE) relate within an attraction-repulsion spectrum?
• RQ2 What embedding characteristics arise when increasing or decreasing attractive forces (exaggeration) in t-SNE?
• RQ3 Can UMAP and FA2 be characterized as particular points on the t-SNE spectrum, and what explains their positions?
• RQ4 What is the impact of optimization choices (negative sampling in UMAP, edge repulsion in FA2) on the effective repulsion/attraction balance?
• RQ5 How does the spectrum trade off between representing continuous manifold structure versus discrete cluster structure across datasets?

主な発見

• There exists an attraction-repulsion spectrum for neighbor embeddings controlled by the exaggeration parameter rho in t-SNE.
• Higher attraction (rho>1) better preserves continuous manifold structure, while higher repulsion (lower rho) emphasizes discrete clusters and increases k-NN recall distortion.
• UMAP embeddings resemble t-SNE with moderate attraction (rho around 4), and ForceAtlas2 resembles t-SNE with very high attraction (rho around 30).
• Increasing rho ultimately leads to embeddings akin to Laplacian eigenmaps in the limit.
• UMAP’s negative sampling reduces effective repulsion, explaining why its layout differs from the raw cross-entropy loss; gamma and m control the repulsion strength.
• k-NN recall declines monotonically with increasing rho, indicating a trade-off between global structure and local neighborhood preservation across the spectrum.
• Across multiple datasets (MNIST, brain organoids, other image datasets), distance correlations between UMAP/FA2 and t-SNE embeddings peak at characteristic rho ranges (UMAP ~4, FA2 ~30).

Figure 3: UMAP with various simplifications. MNIST data set. (a) Default UMAP with $a\approx 1.6$ and $b\approx 0.9$ and LE initialization. (b) UMAP with $a=b=1$ and PCA initialization, the default choice for our experiments. (c) The same as in (b), but using binary $k$ NN affinities ( $v_{ij}=1$ if