[Paper Review] Locally Linear Embedding and its Variants: Tutorial and Survey

Benyamin Ghojogh, Ali Ghodsi+2 more|arXiv (Cornell University)|Nov 22, 2020

Face and Expression Recognition82 references30 citations

TL;DR

A comprehensive tutorial and survey of Locally Linear Embedding (LLE) and its variants, covering theory, extensions, supervised/semi-supervised approaches, robustness, and data fusion.

ABSTRACT

This is a tutorial and survey paper for Locally Linear Embedding (LLE) and its variants. The idea of LLE is fitting the local structure of manifold in the embedding space. In this paper, we first cover LLE, kernel LLE, inverse LLE, and feature fusion with LLE. Then, we cover out-of-sample embedding using linear reconstruction, eigenfunctions, and kernel mapping. Incremental LLE is explained for embedding streaming data. Landmark LLE methods using the Nystrom approximation and locally linear landmarks are explained for big data embedding. We introduce the methods for parameter selection of number of neighbors using residual variance, Procrustes statistics, preservation neighborhood error, and local neighborhood selection. Afterwards, Supervised LLE (SLLE), enhanced SLLE, SLLE projection, probabilistic SLLE, supervised guided LLE (using Hilbert-Schmidt independence criterion), and semi-supervised LLE are explained for supervised and semi-supervised embedding. Robust LLE methods using least squares problem and penalty functions are also introduced for embedding in the presence of outliers and noise. Then, we introduce fusion of LLE with other manifold learning methods including Isomap (i.e., ISOLLE), principal component analysis, Fisher discriminant analysis, discriminant LLE, and Isotop. Finally, we explain weighted LLE in which the distances, reconstruction weights, or the embeddings are adjusted for better embedding; we cover weighted LLE for deformed distributed data, weighted LLE using probability of occurrence, SLLE by adjusting weights, modified LLE, and iterative LLE.

Motivation & Objective

• Present the core idea of LLE as preserving local structure during manifold embedding.
• Survey kernel LLE, inverse LLE, and feature fusion with LLE to broaden applicability.
• Explain out-of-sample embeddings, incremental and landmark LLE for big data and streaming scenarios.
• Detail parameter selection strategies for neighborhood size and other hyperparameters.
• Cover supervised, semi-supervised, and robust LLE variants to handle labels, noise, and outliers.

Proposed method

• Describe the three-step LLE process: construct k-NN graph, compute reconstruction weights with a linear combination of neighbors, and embed using the same weights.
• Derive the closed-form solution for reconstruction weights via a constrained quadratic form.
• Extend to kernel LLE by performing steps in the feature space and deriving kernel-based weights.
• Explain out-of-sample extensions using linear reconstruction, eigenfunctions, and kernel mappings.
• Introduce incremental and landmark LLE for scalability with streaming and large datasets.
• Discuss weighted LLE and fusion with Isomap and other methods for enhanced embeddings.

Experimental results

Research questions

• RQ1 How can LLE be extended to kernel spaces and feature fusion to improve nonlinear manifold embedding?
• RQ2 What are effective out-of-sample extensions, incremental methods, and landmark strategies for large-scale or streaming data?
• RQ3 How can supervised and semi-supervised variants of LLE be formulated and optimized?
• RQ4 What approaches provide robustness to noise and outliers in LLE-based embeddings?
• RQ5 How can LLE be integrated with other manifold learning methods and weighted variants for improved performance?

Key findings

• LLE preserves local structure by learning reconstruction weights from k-NN and reusing them in the embedding space.
• Kernel LLE extends the method to high-dimensional feature spaces by performing reconstruction in the kernel space.
• Out-of-sample extensions include linear reconstruction, eigenfunction-based mapping, and kernel-based mappings, enabling new data to be embedded.
• Incremental and landmark LLE enable embedding of streaming and very large datasets using online or Nyström-based strategies.
• Supervised, semi-supervised, and robust variants address labeled data guidance and noisy/outlier-prone settings.
• Fusion of LLE with Isomap, PCA, and discriminant techniques yields hybrid methods like ISOLLE and discriminant LLE.