[Paper Review] Parametric t-Distributed Stochastic Exemplar-centered Embedding

Martin Renqiang Min, Hongyu Guo+1 more|arXiv (Cornell University)|Oct 14, 2017

Video Surveillance and Tracking Methods17 references3 citations

TL;DR

This paper proposes Parametric t-Distributed Stochastic Exemplar-centered Embedding (pt-SEE), a scalable, robust parametric dimensionality reduction method that replaces pairwise data comparisons in t-SNE with comparisons to a small set of precomputed high-dimensional exemplars. By using exemplars to model local neighborhoods and a shallow network with high-order feature interactions, the method achieves linear computational complexity, eliminates batch size and perplexity sensitivity, and outperforms pt-SNE in visualization quality, robustness, and quantitative metrics on benchmark datasets like MNIST and Fashion-MNIST.

ABSTRACT

Parametric embedding methods such as parametric t-SNE (pt-SNE) have been widely adopted for data visualization and out-of-sample data embedding without further computationally expensive optimization or approximation. However, the performance of pt-SNE is highly sensitive to the hyper-parameter batch size due to conflicting optimization goals, and often produces dramatically different embeddings with different choices of user-defined perplexities. To effectively solve these issues, we present parametric t-distributed stochastic exemplar-centered embedding methods. Our strategy learns embedding parameters by comparing given data only with precomputed exemplars, resulting in a cost function with linear computational and memory complexity, which is further reduced by noise contrastive samples. Moreover, we propose a shallow embedding network with high-order feature interactions for data visualization, which is much easier to tune but produces comparable performance in contrast to a deep neural network employed by pt-SNE. We empirically demonstrate, using several benchmark datasets, that our proposed methods significantly outperform pt-SNE in terms of robustness, visual effects, and quantitative evaluations.

Motivation & Objective

• To address the instability and high computational cost of pt-SNE, which suffers from conflicting batch size and perplexity requirements.
• To eliminate the need for expensive pairwise comparisons in t-SNE by replacing them with comparisons to a small set of representative exemplars.
• To develop a scalable, robust parametric embedding method that supports out-of-sample data inference without retraining.
• To improve usability by simplifying hyperparameter tuning, especially for perplexity and batch size, which are highly sensitive in existing parametric methods.
• To achieve state-of-the-art visualization performance with a simpler, more tunable network architecture than deep feedforward networks used in pt-SNE.

Proposed method

• The method introduces a novel objective function that computes similarity between data points and a fixed set of high-dimensional exemplars instead of pairwise data points, reducing complexity from quadratic to linear in the number of exemplars.
• Exemplars are selected via a few iterations of k-means clustering, capturing both local density and global clustering structure in the data.
• A shallow neural network with high-order feature interactions is used to learn the embedding mapping, replacing the deep feedforward network in pt-SNE for easier tuning and comparable performance.
• Noise contrastive sampling is applied to avoid comparing every data point with all exemplars, further reducing computational and memory costs.
• The perplexity hyperparameter is decoupled from batch size, as it now controls only the expected number of neighboring exemplars globally, not per mini-batch.
• The method enables efficient out-of-sample inference by leveraging the learned parametric mapping function, avoiding costly optimization or approximation.

Experimental results

Research questions

• RQ1 Can a parametric embedding method be designed to avoid the quadratic computational cost of pairwise comparisons in t-SNE while preserving local neighborhood structure?
• RQ2 Does replacing pairwise comparisons with exemplar-based comparisons lead to more robust and scalable embedding with reduced sensitivity to batch size and perplexity?
• RQ3 Can a shallow neural network with high-order feature interactions achieve comparable or better visualization performance than deep networks used in pt-SNE?
• RQ4 To what extent does exemplar-based embedding preserve local and global data structure compared to traditional t-SNE and pt-SNE?
• RQ5 How does the proposed method perform quantitatively and qualitatively on standard benchmark datasets in terms of clustering accuracy and neighborhood preservation?

Key findings

• On the MNIST dataset, dt-see achieved a 1NN test error rate of 8.74% ± 0.37%, significantly outperforming kernel t-SNE (14.2%) and fisher kernel t-SNE (13.7%).
• The quality score for dt-see reached 23.17% at neighborhood size 100, compared to 18.23% for pt-SNE, indicating superior neighborhood preservation.
• hot-see achieved a 1NN test error rate of 9.11% ± 0.43% on MNIST, outperforming pt-SNE’s 12.55% test error rate.
• The method demonstrated robustness to hyperparameter choices: perplexity and batch size had minimal impact on performance due to decoupling of these parameters.
• The exemplar-centered approach reduced computational and memory complexity from quadratic to linear in the number of exemplars, enabling scalability to large datasets.
• The shallow network with high-order feature interactions achieved performance comparable to deep networks in pt-SNE but with significantly easier hyperparameter tuning.