[Paper Review] DID: Distributed Incremental Block Coordinate Descent for Nonnegative Matrix Factorization

Tianxiang Gao, Chris Chu|arXiv (Cornell University)|Feb 25, 2018

Matrix Theory and Algorithms4 citations

TL;DR

This paper proposes DID, a distributed incremental block coordinate descent algorithm for nonnegative matrix factorization (NMF) that enables efficient, scalable NMF in distributed memory systems. By updating the factor matrices incrementally using the most recent residual and requiring only one communication step per iteration, DID achieves faster convergence and linear scalability compared to existing methods, especially on large-scale datasets with up to 10^8 samples.

ABSTRACT

Nonnegative matrix factorization (NMF) has attracted much attention in the last decade as a dimension reduction method in many applications. Due to the explosion in the size of data, naturally the samples are collected and stored distributively in local computational nodes. Thus, there is a growing need to develop algorithms in a distributed memory architecture. We propose a novel distributed algorithm, called extit{distributed incremental block coordinate descent} (DID), to solve the problem. By adapting the block coordinate descent framework, closed-form update rules are obtained in DID. Moreover, DID performs updates incrementally based on the most recently updated residual matrix. As a result, only one communication step per iteration is required. The correctness, efficiency, and scalability of the proposed algorithm are verified in a series of numerical experiments.

Motivation & Objective

• Address the challenge of scaling nonnegative matrix factorization (NMF) to massive datasets that cannot fit in a single machine’s memory.
• Overcome the limitations of centralized NMF algorithms by designing a distributed algorithm suitable for modern cluster architectures.
• Reduce communication overhead in distributed NMF by minimizing the number of synchronization steps per iteration.
• Achieve faster convergence and linear scalability on large-scale synthetic and real-world datasets compared to existing distributed NMF methods.

Proposed method

• Distribute the data matrix X and factor matrix C across computational nodes by splitting columns into blocks, enabling parallel updates of C.
• Use an incremental update strategy for the basis matrix B by leveraging the most recently updated residual matrix, improving convergence speed.
• Derive closed-form update rules for B and C using block coordinate descent, ensuring computational efficiency.
• Implement the algorithm using Message Passing Interface (MPI) without requiring a master processor for synchronization, enhancing scalability.
• Limit communication to a single step per iteration by propagating only necessary intermediate variables (e.g., δb), reducing network overhead.
• Apply well-scaled initial values and K-means initialization to prevent numerical instability from near-zero norms in B or C.

Experimental results

Research questions

• RQ1 Can a distributed NMF algorithm be designed to achieve fast convergence with minimal communication overhead in large-scale settings?
• RQ2 How does incremental updating of the basis matrix B using the most recent residual improve convergence compared to batch or sequential updates?
• RQ3 To what extent does the proposed algorithm scale linearly with increasing data size and number of computational nodes?
• RQ4 How does the performance of the proposed algorithm compare to existing distributed NMF methods like D-HALS, HPC-ANLS, and Maxios in terms of convergence speed and communication cost?
• RQ5 Can the algorithm maintain numerical stability and correctness when applied to both sparse and dense real-world datasets?

Key findings

• DID converges faster than HALS, BCD, DBCD, and other distributed NMF methods in terms of runtime, achieving up to 10–15% speedup over DBCD on 16 nodes.
• The algorithm exhibits linear scalability with respect to the number of samples, with runtime increasing proportionally as N increases from 10^5 to 10^8.
• Only one communication step per iteration is required, significantly reducing communication overhead compared to methods like HPC-ANLS (6 steps) and Maxios (8 steps).
• On sparse datasets like MNIST and 20News, HPC-ANLS performs better due to its exact NNLS subproblem solving, but DID remains competitive and scalable.
• DID maintains correctness by matching the number of iterations with BCD and DBCD, confirming that its update rules are mathematically equivalent.
• The algorithm is robust across both synthetic and real-world datasets, including dense (UMist, YaleB) and sparse (MNIST, 20News) data, with consistent performance.