[Paper Review] Generic Multiplicative Methods for Implementing Machine Learning Algorithms on MapReduce

Song Liu, Peter Flach+1 more|arXiv (Cornell University)|Nov 9, 2011

Stochastic Gradient Optimization Techniques7 references3 citations

TL;DR

This paper proposes a generic multiplicative model for scaling machine learning algorithms on the MapReduce framework, enabling efficient parallelization of similarity-based and iterative algorithms like NMF, SVM, and PageRank. By abstracting common computational patterns—particularly matrix multiplication and multiplicative updates—it achieves linear speedup and high scalability across large datasets, outperforming traditional O(m³) implementations in both runtime and efficiency.

ABSTRACT

In this paper we introduce a generic model for multiplicative algorithms which is suitable for the MapReduce parallel programming paradigm. We implement three typical machine learning algorithms to demonstrate how similarity comparison, gradient descent, power method and other classic learning techniques fit this model well. Two versions of large-scale matrix multiplication are discussed in this paper, and different methods are developed for both cases with regard to their unique computational characteristics and problem settings. In contrast to earlier research, we focus on fundamental linear algebra techniques that establish a generic approach for a range of algorithms, rather than specific ways of scaling up algorithms one at a time. Experiments show promising results when evaluated on both speedup and accuracy. Compared with a standard implementation with computational complexity $O(m^3)$ in the worst case, the large-scale matrix multiplication experiments prove our design is considerably more efficient and maintains a good speedup as the number of cores increases. Algorithm-specific experiments also produce encouraging results on runtime performance.

Motivation & Objective

• To develop a generic, reusable framework for scaling multiple machine learning algorithms on the MapReduce parallel computing model.
• To identify and abstract common computational patterns—especially multiplicative updates and matrix operations—across diverse algorithms like NMF, SVM, and PageRank.
• To improve scalability and performance of similarity-based and iterative learning algorithms on large-scale data by leveraging partitioning and parallelization strategies.
• To demonstrate that a unified model can outperform individual algorithm-specific optimizations in terms of efficiency and general applicability.
• To establish a foundation for future large-scale learning systems by enabling efficient, distributed execution of iterative and similarity-based algorithms.

Proposed method

• Design a generic multiplicative model that abstracts iterative update rules common in NMF, SVM, and PageRank as matrix multiplication operations.
• Implement two variants of large-scale matrix multiplication tailored to sparse and dense matrix settings, using data partitioning to balance load and maximize parallelism.
• Introduce a flexible 'Partition Schema' configuration to adapt to different matrix densities and cluster scales, optimizing performance across diverse workloads.
• Apply the model to three core algorithms: NMF via multiplicative updates, SVM using kernel matrix computation, and PageRank via iterative matrix-vector multiplication.
• Use MapReduce's functional abstraction to express all operations as map and reduce phases, enabling distributed execution on Hadoop-like clusters.
• Optimize computation by minimizing redundant data shuffling and aligning matrix partitions with computational access patterns.

Experimental results

Research questions

• RQ1 Can a single generic model effectively abstract and scale multiple machine learning algorithms with iterative, multiplicative updates?
• RQ2 How does the proposed multiplicative model achieve efficient parallelization on the MapReduce paradigm for large-scale data?
• RQ3 What partitioning and load-balancing strategies maximize performance across different matrix densities and cluster sizes?
• RQ4 To what extent does the generic model maintain accuracy while improving runtime and scalability compared to standard implementations?
• RQ5 Can the model be applied to diverse algorithms like NMF, SVM, and PageRank with consistent performance gains?

Key findings

• The generic multiplicative model achieved linear speedup in large-scale matrix multiplication, significantly outperforming theoretical O(m³) complexity in practice.
• The implementation of NMF showed strong speedup, especially in similarity comparison and multiplicative update phases, due to efficient partitioning and parallelization.
• The SVM implementation achieved comparable performance to LibSVM on large kernel matrices, demonstrating good scalability on distributed systems.
• PageRank was successfully applied to a Wikipedia corpus of 5.7 million pages, converging in 100 iterations within one hour, with empirically sound ranking results.
• Matrix density was identified as a critical factor influencing speedup rates, with better performance observed in sparse settings due to reduced communication overhead.
• The proposed 'Partition Schema' configuration significantly improved performance by adapting to matrix sparsity and cluster scale, enabling efficient load balancing.