[Paper Review] Divide-and-Conquer Matrix Factorization

Lester Mackey, Ameet Talwalkar+1 more|arXiv (Cornell University)|Jul 5, 2011

Sparse and Compressive Sensing Techniques34 references18 citations

TL;DR

This paper proposes Divide-Factor-Combine (DFC), a parallel divide-and-conquer framework for large-scale noisy matrix factorization. It splits the matrix into subproblems solved in parallel using any base factorization algorithm, then combines results via randomized matrix approximation techniques, achieving near-linear to super-linear speed-ups while maintaining high-probability recovery guarantees comparable to the base algorithm.

ABSTRACT

This work introduces Divide-Factor-Combine (DFC), a parallel divide-and-conquer framework for noisy matrix factorization. DFC divides a large-scale matrix factorization task into smaller subproblems, solves each subproblem in parallel using an arbitrary base matrix factorization algorithm, and combines the sub-problem solutions using techniques from randomized matrix approximation. Our experiments with collaborative filtering, video background modeling, and simulated data demonstrate the near-linear to super-linear speed-ups attainable with this approach. Moreover, our analysis shows that DFC enjoys high-probability recovery guarantees comparable to those of its base algorithm.

Motivation & Objective

• To address the computational bottleneck of large-scale matrix factorization in applications like collaborative filtering and video background modeling.
• To enable efficient parallelization of matrix factorization without sacrificing solution quality under noisy conditions.
• To develop a framework that maintains strong theoretical recovery guarantees while scaling to large matrices.
• To combine subproblem solutions effectively using randomized matrix approximation techniques for global consistency.

Proposed method

• The matrix is partitioned into smaller submatrices to enable parallel processing.
• Each submatrix is factorized independently using an arbitrary base matrix factorization algorithm.
• Subproblem solutions are combined using randomized matrix approximation to reconstruct a global low-rank approximation.
• The framework leverages theoretical results from randomized matrix approximation to ensure stability and accuracy in the combined solution.
• The approach is designed to be modular, allowing integration with any existing matrix factorization algorithm as the base solver.

Experimental results

Research questions

• RQ1 Can a divide-and-conquer strategy achieve near-linear or super-linear speed-ups in large-scale matrix factorization under noise?
• RQ2 Does the combination of subproblem solutions preserve the high-probability recovery guarantees of the base factorization algorithm?
• RQ3 How effective is the randomized matrix approximation technique in combining sub-solutions without significant error accumulation?
• RQ4 To what extent does the framework scale across diverse applications such as collaborative filtering and video background modeling?

Key findings

• DFC achieves near-linear to super-linear speed-ups on collaborative filtering, video background modeling, and simulated data workloads.
• The framework maintains high-probability recovery guarantees that are comparable to those of the base matrix factorization algorithm.
• The use of randomized matrix approximation enables accurate and stable combination of sub-solutions across diverse data types.
• Empirical results demonstrate consistent performance gains across multiple real-world and synthetic datasets.