[Paper Review] Multidimensional Scaling for Big Data

Pedro Delicado, Cristian Pachon-Garcia|arXiv (Cornell University)|Jul 23, 2020

Face and Expression Recognition7 references4 citations

TL;DR

This paper proposes two new algorithms—interpolation MDS and divide-and-conquer MDS—for scalable multidimensional scaling (MDS) on big data, leveraging Gower’s interpolation formula and data partitioning to reduce memory and computation costs. The authors demonstrate that interpolation MDS is the fastest method with high accuracy, outperforming existing approaches in both speed and statistical fidelity on large-scale datasets like EMNIST with over 800,000 points.

ABSTRACT

We present a set of algorithms implementing multidimensional scaling (MDS) for large data sets. MDS is a family of dimensionality reduction techniques using a $n imes n$ distance matrix as input, where $n$ is the number of individuals, and producing a low dimensional configuration: a $n imes r$ matrix with $r<

Motivation & Objective

• Address the computational infeasibility of classical MDS for large datasets due to O(n³) time and O(n²) memory requirements.
• Develop efficient, scalable MDS algorithms that maintain high accuracy while reducing memory and time complexity.
• Compare multiple non-standard MDS algorithms, including novel proposals, to identify the most effective approach for big data.
• Implement and evaluate the algorithms on both simulated data and a real-world large dataset (EMNIST) to validate performance and scalability.
• Create an R package (bigmds) to provide accessible, production-ready tools for researchers and practitioners working with large-scale MDS.

Proposed method

• Propose interpolation MDS, a novel method that uses Gower’s interpolation formula to project non-landmark points onto a low-dimensional configuration derived from landmark points, avoiding distance matrix storage issues.
• Introduce divide-and-conquer MDS, which partitions the dataset into smaller subsets, applies classical MDS to each, and combines results via Procrustes transformation to align configurations.
• Reveal that the triangulation method in landmark MDS (LMDS) is mathematically equivalent to Gower’s interpolation formula, unifying two previously distinct approaches.
• Implement and benchmark six MDS algorithms: LMDS, interpolation MDS, RMDS, pivot MDS, divide-and-conquer MDS, and fast MDS, using consistent simulation and real-data settings.
• Use a simulation study with known low-dimensional structures to assess accuracy via correlation with classical MDS and variance estimation.
• Apply all algorithms to the EMNIST dataset (n > 800,000) to test real-world scalability and performance, with no gold standard available for comparison.

Experimental results

Research questions

• RQ1 Can interpolation MDS and divide-and-conquer MDS achieve high accuracy while significantly reducing computational and memory costs compared to classical MDS for big data?
• RQ2 How do the proposed algorithms compare in terms of computational efficiency and statistical accuracy when applied to large-scale datasets?
• RQ3 Is the triangulation method used in landmark MDS mathematically equivalent to Gower’s interpolation formula, and what are the implications for algorithm design?
• RQ4 How do the six MDS algorithms perform on real-world big data, such as the EMNIST dataset, in terms of execution time and visual configuration quality?
• RQ5 Which algorithm offers the best trade-off between speed, accuracy, and memory efficiency for large-scale MDS applications?

Key findings

• All six algorithms produce MDS configurations highly correlated with classical MDS (correlations > 0.98), indicating strong preservation of underlying low-dimensional structure.
• Interpolation MDS is the fastest algorithm in simulations, followed closely by LMDS and pivot MDS, with execution times under 100 seconds for the simulation setup.
• RMDS is the slowest algorithm in simulations, requiring approximately 10 minutes to complete, while fast MDS and divide-and-conquer MDS take around 3.5 minutes.
• On the EMNIST dataset (n > 800,000), all algorithms completed within 3.5 minutes except RMDS, which took 10.3 minutes, confirming scalability for real-world big data.
• Pivot MDS, LMDS, and interpolation MDS were the fastest on EMNIST, each completing in approximately 1.5 minutes, demonstrating strong real-world performance.
• Interpolation MDS is recommended as the optimal choice due to its speed, high accuracy (indistinguishable from LMDS and RMDS), and avoidance of large intermediate distance matrices.