[Paper Review] Expanded Alternating Optimization of Nonconvex Functions with Applications to Matrix Factorization and Penalized Regression

William J. Murdoch, Mu Zhu|arXiv (Cornell University)|Dec 12, 2014

Sparse and Compressive Sensing Techniques12 references3 citations

TL;DR

This paper proposes an expanded alternating optimization (EAO) framework to improve nonconvex optimization in matrix factorization and penalized regression by conducting additional searches over novel, problem-specific subspaces after standard alternating optimization. The method significantly enhances convergence and solution quality, especially when tailored to data structure, with demonstrated improvements on real-world and simulated datasets using MC+ and SparseNet penalties.

ABSTRACT

We propose a general technique for improving alternating optimization (AO) of nonconvex functions. Starting from the solution given by AO, we conduct another sequence of searches over subspaces that are both meaningful to the optimization problem at hand and different from those used by AO. To demonstrate the utility of our approach, we apply it to the matrix factorization (MF) algorithm for recommender systems and the coordinate descent algorithm for penalized regression (PR), and show meaningful improvements using both real-world (for MF) and simulated (for PR) data sets. Moreover, we demonstrate for MF that, by constructing search spaces customized to the given data set, we can significantly increase the convergence rate of our technique.

Motivation & Objective

• Address the limitation of standard alternating optimization (AO) in nonconvex problems, which often converges to poor local minima or saddle points.
• Improve solution quality and convergence speed in matrix factorization for recommender systems by exploring subspaces beyond standard AO updates.
• Enhance coordinate descent performance in penalized regression with nonconvex penalties (e.g., MC+), particularly in high-dimensional settings.
• Develop a systematic method to explore alternative search directions that are meaningful to the problem and distinct from AO’s original blocks.
• Demonstrate that customizing search subspaces to data characteristics can dramatically accelerate convergence and improve final objective values.

Proposed method

• Propose Expanded Alternating Optimization (EAO): after standard AO converges, perform a second phase of optimization over new, problem-specific subspaces not used in AO.
• Define new subspaces based on structural properties of the problem—e.g., low-rank structure in matrix factorization or sparsity patterns in regression.
• For matrix factorization, search over subspaces defined by user or item latent vectors that are orthogonal to the current AO solution direction.
• For penalized regression with MC+ penalty, explore subspaces defined by active sets and sign patterns of coefficients, using piecewise-linear solutions to the first-order conditions.
• Use a case-based strategy to solve for optimal variable updates: test multiple (ψ, ξ) pairs corresponding to different sign and magnitude regimes of βj, and discard inconsistent solutions.
• On each interval defined by critical thresholds (e.g., γλ), solve a linear equation in v to find candidate solutions, selecting the one minimizing the objective function.

Experimental results

Research questions

• RQ1 Can expanding the search space beyond standard alternating optimization improve convergence and solution quality in nonconvex matrix factorization?
• RQ2 How does the performance of EAO compare to standard AO and coordinate descent in penalized regression with nonconvex penalties like MC+?
• RQ3 To what extent can customizing the search subspaces to data-specific structure accelerate convergence in EAO?
• RQ4 What is the impact of exploring multiple solution branches (based on sign and magnitude regimes) on the final objective value?
• RQ5 Can EAO consistently escape poor local optima or saddle points that standard AO gets trapped in?

Key findings

• EAO significantly improves the objective value in matrix factorization on real-world recommender system data, outperforming standard AO and coordinate descent.
• In penalized regression with MC+ penalty, EAO achieves better solution quality than standard coordinate descent, particularly in high-dimensional settings.
• When search subspaces are customized to the data (e.g., based on sparsity or low-rank structure), EAO converges much faster than generic implementations.
• The method successfully escapes poor local optima by exploring alternative subspaces, with consistent improvements observed across multiple real-world and simulated datasets.
• The piecewise-linear solution strategy for MC+ penalty allows exact and efficient computation of candidate solutions across different regimes of βj.
• Multiple consistent solutions may exist; the method selects the one minimizing the objective, ensuring improved convergence over standard AO.