[Paper Review] Algorithms for solving optimization problems arising from deep neural net models: smooth problems

Vyacheslav Kungurtsev, Tomáš Pevný|arXiv (Cornell University)|Jun 30, 2018

Sparse and Compressive Sensing Techniques20 references5 citations

TL;DR

This paper proposes a Lanczos-based Newton-negative curvature method for optimizing deep neural networks by explicitly leveraging directions of negative curvature to escape saddle points, significantly outperforming stochastic gradient descent (SGD) in convergence speed and objective value reduction on a security anomaly detection dataset, with empirical results showing superior performance despite higher per-iteration cost.

ABSTRACT

Machine Learning models incorporating multiple layered learning networks have been seen to provide effective models for various classification problems. The resulting optimization problem to solve for the optimal vector minimizing the empirical risk is, however, highly nonlinear. This presents a challenge to application and development of appropriate optimization algorithms for solving the problem. In this paper, we summarize the primary challenges involved and present the case for a Newton-based method incorporating directions of negative curvature, including promising numerical results on data arising from security anomally deetection.

Motivation & Objective

• To address the challenge of slow convergence in deep neural network optimization due to saddle points, which hinder first-order methods like SGD.
• To investigate whether explicitly incorporating directions of negative curvature can improve convergence and objective function reduction in nonconvex deep learning problems.
• To develop a practical, scalable second-order optimization method that avoids the pitfalls of standard Newton methods by leveraging Hessian-vector products and Lanczos iterations.
• To evaluate the effectiveness of deterministic mini-batch cycling versus random selection in maintaining convergence when using negative curvature directions.
• To explore the feasibility of using smaller batch sizes for Hessian-vector product computation to reduce computational cost while preserving useful curvature information.

Proposed method

• Uses the Lanczos algorithm to compute approximate Newton directions and identify directions of negative curvature from the Hessian at the current iterate.
• Employs Hessian-vector products approximated via finite differences or automatic differentiation to enable Hessian-free computation.
• Combines the Newton direction (s) and negative curvature direction (d) into a combined search direction t = s + d to ensure descent in both positive and negative curvature regions.
• Applies a line search on a mini-batch function to determine the step size α, ensuring sufficient decrease in the objective function.
• Implements a deterministic round-robin index selection for mini-batches, which was found essential for convergence, unlike random selection.
• Uses a fixed number of Lanczos iterations (q ≈ 5) based on empirical optimization of performance and cost.

Experimental results

Research questions

• RQ1 Can explicit use of negative curvature directions significantly improve convergence in deep neural network optimization compared to first-order methods?
• RQ2 Does combining Newton and negative curvature directions lead to faster objective reduction in nonconvex, smooth optimization problems arising from DNNs?
• RQ3 How does deterministic mini-batch cycling compare to random selection in maintaining convergence when using second-order information in stochastic optimization?
• RQ4 Can smaller batch sizes be used effectively for Hessian-vector product computation without degrading optimization performance?
• RQ5 Is the Lanczos-based method robust and scalable for real-world, large-scale machine learning problems such as security anomaly detection?

Key findings

• The proposed Lanczos Newton-negative curvature method significantly outperforms SGD in reducing the objective function value over 1000 iterations on a private security anomaly detection dataset from Cisco.
• Despite higher computational cost per iteration, the method achieves faster convergence and lower objective values than SGD with constant, diminishing, or line-search step sizes.
• Deterministic cycling of mini-batch indices was essential for convergence; random selection led to objective function ascent, suggesting overfitting to individual batches.
• The method successfully leveraged negative curvature directions to escape saddle points, demonstrating that such directions provide a reliable and effective descent path.
• Empirical results showed that combining Newton and negative curvature directions (t = s + d) was effective, with no significant advantage found for more complex combinations.
• The use of finite differences or automatic differentiation for Hessian-vector products enabled practical implementation without explicit Hessian storage.