[Paper Review] Parallel Stochastic Newton Method

Mojmír Mutný, Peter Richtárik|Edinburgh Research Explorer|May 4, 2017

Stochastic Gradient Optimization Techniques23 references3 citations

TL;DR

This paper proposes a parallel stochastic Newton method (PSN) for minimizing smooth, strongly convex functions, leveraging curvature information through randomized, overlapping sub-matrix sampling to accelerate convergence. The method achieves theoretical and empirical speedups over serial and parallel coordinate descent methods (PCDM), particularly in big data settings like empirical risk minimization.

ABSTRACT

We propose a parallel stochastic Newton method (PSN) for minimizing unconstrained smooth convex functions. We analyze the method in the strongly convex case, and give conditions under which acceleration can be expected when compared to its serial counterpart. We show how PSN can be applied to the empirical risk minimization problem, and demonstrate the practical efficiency of the method through numerical experiments and models of simple matrix classes.

Motivation & Objective

• Address the inefficiency of classical second-order methods in big data by designing a scalable, parallelizable algorithm that retains curvature information.
• Overcome the computational bottleneck of full Hessian inversion in Newton methods by using randomized, overlapping sub-matrix approximations.
• Provide theoretical convergence guarantees for the parallel stochastic Newton method under strong convexity and smoothness assumptions.
• Demonstrate superior performance compared to existing methods, particularly the parallel coordinate descent method (PCDM), in both theory and practice.
• Extend the method to empirical risk minimization (ERM) problems by dual reformulation and apply it to statistical models like ridge regression.

Proposed method

• Proposes a parallel stochastic Newton method (PSN) that samples overlapping blocks of the Hessian approximation matrix to exploit curvature information.
• Uses randomized sampling of index sets $ S_k $ to form sub-matrices $ \mathbf{X}_{S_k} $, enabling parallel updates across multiple blocks.
• Employs a dual formulation of ERM problems to derive a dual objective where PSN can be applied directly, with $ \alpha $-updates based on dual gradients and Hessian approximations.
• Performs primal-dual updates: first compute $ w^k = \nabla g^*(\bar{\alpha}_k) $, then solve sub-problems $ h^{k_j} = \arg\min_h \langle I_{S_j^k}\mathbf{A}^\top w^k, h \rangle + \frac{1}{2}\langle h, \mathbf{X}_{S_j^k} h \rangle + \sum_{i\in S_j^k} \nabla\psi_i(\alpha_i^k) h_i $.
• Combines multiple sub-problem solutions via averaging: $ \alpha^{k+1} = \alpha^k + \frac{1}{b} \sum_{j=1}^c h^{k_j} $, with dual averaging for $ \bar{\alpha}^{k+1} $.
• The Hessian approximation matrix $ \mathbf{X} $ is defined as $ \frac{1}{\lambda n^2}\mathbf{A}^\top\mathbf{A} + \frac{\mathbf{D}(\gamma)^{-1}}{n} $, combining data and regularizer curvature.

Experimental results

Research questions

• RQ1 Can a parallel stochastic Newton method achieve theoretical acceleration over its serial counterpart in strongly convex optimization?
• RQ2 How does the choice of overlapping, non-fixed sampling of Hessian blocks affect convergence compared to fixed, non-overlapping blocks?
• RQ3 In what settings does PSN outperform the parallel coordinate descent method (PCDM) in terms of convergence rate and practical efficiency?
• RQ4 What are the theoretical conditions under which PSN achieves speedup in big data optimization problems?
• RQ5 How can the PSN framework be adapted to empirical risk minimization (ERM) problems with smooth, strongly convex objectives?

Key findings

• PSN achieves theoretical convergence with linear rate under strong convexity, and the convergence rate depends on the spectral properties of the Hessian approximation matrix $ \mathbf{X} $.
• The method demonstrates acceleration over the serial stochastic Newton method due to parallelization, with speedup confirmed in special matrix classes like $ \rho $-matrices and $ \alpha $-tridiagonal matrices.
• For $ \rho $-matrices, PSN achieves a convergence rate that improves upon PCDM, particularly when the sampling distribution is well-chosen.
• In numerical experiments, PSN outperforms PCDM in both convergence speed and final objective value, especially in high-dimensional settings.
• The dual formulation of ERM allows direct application of PSN, with convergence guarantees transferred from the main theorem by substituting $ \mathbf{M} \leftarrow \mathbf{X} $.
• The authors conjecture that the duality gap bound from prior work [12] likely holds for PSN, though a formal proof is left for future work.