[Paper Review] Low-rank Interaction with Sparse Additive Effects Model for Large Data Frames

Geneviève Robin, Hoi-To Wai+3 more|arXiv (Cornell University)|Dec 20, 2018

CCD and CMOS Imaging Sensors4 citations

TL;DR

This paper proposes the Low-rank Interaction with Sparse Additive Effects (LORIS) model, a convex, doubly penalized quasi-maximum likelihood framework that jointly estimates low-rank interactions and sparse main effects (e.g., row, column, covariate effects) in large, heterogeneous, incomplete data frames. It introduces a Mixed Coordinate Gradient Descent (MCGD) algorithm with provable O(1/ε) sublinear convergence, outperforming preprocessing-based methods in imputation and estimation on simulated and survey data.

ABSTRACT

Many applications of machine learning involve the analysis of large data frames-matrices collecting heterogeneous measurements (binary, numerical, counts, etc.) across samples-with missing values. Low-rank models, as studied by Udell et al. [30], are popular in this framework for tasks such as visualization, clustering and missing value imputation. Yet, available methods with statistical guarantees and efficient optimization do not allow explicit modeling of main additive effects such as row and column, or covariate effects. In this paper, we introduce a low-rank interaction and sparse additive effects (LORIS) model which combines matrix regression on a dictionary and low-rank design, to estimate main effects and interactions simultaneously. We provide statistical guarantees in the form of upper bounds on the estimation error of both components. Then, we introduce a mixed coordinate gradient descent (MCGD) method which provably converges sub-linearly to an optimal solution and is computationally efficient for large scale data sets. We show on simulated and survey data that the method has a clear advantage over current practices, which consist in dealing separately with additive effects in a preprocessing step.

Motivation & Objective

• To address the lack of statistical and computational frameworks that simultaneously model low-rank interactions and main additive effects (e.g., row, column, covariate) in large, heterogeneous, incomplete data frames.
• To overcome limitations of existing methods that either ignore additive effects or require preprocessing steps like mean-centering, which can degrade performance on large-scale heterogeneous data.
• To develop a statistically sound, convex optimization framework with non-asymptotic error bounds for both low-rank and sparse components.
• To design a computationally efficient, scalable algorithm—MCGD—that supports distributed implementation and provably converges to an ε-optimal solution in O(1/ε) iterations.

Proposed method

• The LORIS model combines matrix regression on a dictionary with low-rank approximation, using a convex, doubly penalized quasi-likelihood approach with nuclear norm relaxation for rank and ℓ1-penalty for sparsity.
• The method jointly estimates the low-rank component and sparse additive effects by minimizing a loss function that incorporates heterogeneous exponential family quasi-likelihood for mixed data types (binary, count, continuous).
• The Mixed Coordinate Gradient Descent (MCGD) algorithm uses a proximal update for the sparse component and a conditional gradient (Frank-Wolfe-type) update for the low-rank component, enabling efficient and scalable optimization.
• The algorithm features a mixed update strategy: proximal updates for sparsity and conditional gradient steps for low-rank structure, with step size determined via a line search that aggregates local gradients in a distributed setting.
• For distributed implementation, the method computes top singular vectors and gradients in parallel across data partitions, with only matrix-vector products transmitted to the central server to preserve data privacy.
• The convergence rate is proven to be sublinear at O(1/ε), and the method is shown to be strictly faster than the FW-T method in convergence speed via a novel proof technique.

Experimental results

Research questions

• RQ1 Can a unified statistical model jointly estimate low-rank interactions and sparse additive effects (e.g., row, column, covariate effects) in large, heterogeneous, incomplete data frames with theoretical guarantees?
• RQ2 Does the proposed MCGD algorithm achieve provably faster convergence than existing first-order methods like FW-T, especially in large-scale settings?
• RQ3 How does the performance of the LORIS model compare to standard preprocessing approaches (e.g., mean-centering) in terms of estimation accuracy and missing value imputation quality?
• RQ4 What is the impact of ignoring main effects during preprocessing on downstream analytics in large, heterogeneous data frames?
• RQ5 Can the MCGD algorithm be efficiently distributed across multiple workers without revealing raw data, while maintaining convergence and privacy?

Key findings

• The LORIS model achieves non-asymptotic upper bounds on estimation error for both the low-rank and sparse additive components, providing strong statistical guarantees.
• The MCGD algorithm converges sublinearly to an ε-optimal solution in O(1/ε) iterations, with a convergence rate strictly faster than the FW-T method due to a novel proof technique.
• The method outperforms standard preprocessing heuristics (e.g., mean-centering) in both estimation accuracy and missing value imputation on simulated and real survey data.
• The distributed MCGD implementation achieves O(|Ξ| + max{n,p} log(1/δ)) complexity per iteration at the central server and O(|Ωk|max{n,p} log(1/δ)) per worker, scaling efficiently with data partitioning.
• The algorithm preserves data privacy by transmitting only matrix-vector products, not raw gradients, enabling secure distributed computation.
• Numerical experiments confirm that removing row and column effects via preprocessing leads to severe performance degradation, validating the necessity of joint modeling.