[Paper Review] Pattern Search Multidimensional Scaling

Georgios Paraskevopoulos, Efthymios Tzinis+2 more|arXiv (Cornell University)|Jun 1, 2018

Face and Expression Recognition56 references3 citations

TL;DR

This paper proposes Pattern Search Multidimensional Scaling (PS-MDS), a derivative-free optimization approach for nonlinear manifold learning that replaces gradient descent with pattern search in a fixed-radius neighborhood. It provides theoretical convergence guarantees via the General Pattern Search (GPS) framework and demonstrates state-of-the-art performance on clean and noisy synthetic and real-world datasets, including word embeddings and MNIST, with faster convergence than SMACOF-MDS.

ABSTRACT

We present a novel view of nonlinear manifold learning using derivative-free optimization techniques. Specifically, we propose an extension of the classical multi-dimensional scaling (MDS) method, where instead of performing gradient descent, we sample and evaluate possible "moves" in a sphere of fixed radius for each point in the embedded space. A fixed-point convergence guarantee can be shown by formulating the proposed algorithm as an instance of General Pattern Search (GPS) framework. Evaluation on both clean and noisy synthetic datasets shows that pattern search MDS can accurately infer the intrinsic geometry of manifolds embedded in high-dimensional spaces. Additionally, experiments on real data, even under noisy conditions, demonstrate that the proposed pattern search MDS yields state-of-the-art results.

Motivation & Objective

• To address the limitations of gradient-based MDS methods in non-differentiable or computationally expensive gradient scenarios.
• To develop a derivative-free optimization framework for non-metric MDS that ensures theoretical convergence.
• To improve computational efficiency, convergence speed, and solution accuracy through heuristic enhancements.
• To evaluate PS-MDS on manifold unfolding, word embeddings, and optical digit recognition under clean and noisy conditions.
• To provide open-source code for reproducibility and community adoption.

Proposed method

• The algorithm performs iterative updates by sampling and evaluating candidate moves within a fixed-radius sphere around each point in the embedded space.
• It uses a pattern search strategy that explores directions in the neighborhood without requiring gradient computation.
• The method is formally linked to the General Pattern Search (GPS) framework, enabling fixed-point convergence guarantees.
• Heuristics are introduced to bias search directions toward principal components of local neighborhoods, improving convergence and efficiency.
• The algorithm is optimized for scalability using landmark-based approximations and fast MDS approximations.
• It supports non-symmetric distance metrics, motivated by cognitive science, to better model complex data geometries.

Experimental results

Research questions

• RQ1 Can a derivative-free optimization approach achieve comparable or better performance than gradient-based MDS in nonlinear manifold learning?
• RQ2 Does the pattern search strategy provide theoretical convergence guarantees in the context of non-metric MDS?
• RQ3 How does PS-MDS perform on noisy and corrupted data compared to state-of-the-art MDS and LLE-based methods?
• RQ4 Can heuristic enhancements improve convergence speed and solution quality without sacrificing accuracy?
• RQ5 Can PS-MDS effectively handle both global geometry (like MDS) and local non-convexities (like LLE) by combining their strengths?

Key findings

• PS-MDS achieves state-of-the-art performance on word embedding similarity tasks using MEN and SimLex-999 datasets, even with noisy word vectors.
• On the MNIST dataset, PS-MDS yields KNN classification accuracy comparable to state-of-the-art methods, demonstrating robustness on real-world data.
• In convergence comparisons, PS-MDS reaches similar or better solutions than SMACOF-MDS in fewer epochs, indicating faster convergence.
• On synthetic datasets with holes (non-convex manifolds), PS-MDS shows superior geometric fidelity compared to standard MDS, though LLE variants perform better on highly non-convex shapes.
• PS-MDS outperforms other MDS methods on sparse data, confirming its strength in capturing global geometry over local neighborhoods.
• The open-source implementation enables reproducibility and facilitates future research and integration into existing pipelines.