[Paper Review] A Survey of Optimization Methods from a Machine Learning Perspective

Shiliang Sun, Zehui Cao+2 more|arXiv (Cornell University)|Jun 17, 2019

Machine Learning and Data Classification211 references44 citations

TL;DR

This survey categorizes optimization methods from a machine learning perspective, discusses their advantages, limitations, and suitable application contexts, and highlights challenges and open problems in ML optimization.

ABSTRACT

Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this paper, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Next, we summarize the applications and developments of optimization methods in some popular machine learning fields. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

Motivation & Objective

• Explain optimization problems in machine learning across supervised, semi-supervised, unsupervised, and reinforcement learning.
• Systematically review fundamental and recent optimization methods used in ML.
• Discuss applications of optimization methods in deep neural networks, RL, meta-learning, variational inference, and MCMC.
• Identify challenges and open problems in optimization for ML to guide future research.

Proposed method

• Classify optimization methods by gradient information: first-order, high-order, and derivative-free.
• Discuss preconditioning and its role in accelerating convergence.
• Describe representative algorithms (e.g., SGD and its variants, momentum, Nesterov, AdaGrad/Adam, variance reduction methods).
• Explain the trade-offs between convergence speed, computational cost, and scalability for large-scale ML problems.
• Connect optimization methods to ML applications such as DNNs, RL, meta-learning, variational inference, and MCMC.

Experimental results

Research questions

• RQ1 What are the fundamental and modern optimization methods most relevant to machine learning, and what are their strengths and limitations?
• RQ2 How do optimization techniques perform across key ML paradigms (supervised, unsupervised, reinforcement learning, Bayesian inference) and what configurations are most effective?
• RQ3 What are the remaining challenges and open problems in ML optimization that warrant further research?

Key findings

• First-order methods (e.g., SGD and variants) remain dominant in ML due to scalability, with acceleration and variance reduction techniques improving convergence.
• High-order methods offer faster convergence when curvature information is leveraged, but face practical challenges such as Hessian storage and computation.
• Derivative-free methods are useful when derivatives are unavailable or expensive, and can complement gradient-based approaches.
• Adaptive learning rate methods (AdaGrad, RMSProp, Adam) provide practical performance gains and reduced manual tuning, often outperforming fixed learning-rate strategies.
• Variance reduction techniques (e.g., SAG) can achieve faster convergence by mitigating gradient noise in SGD.
• Optimization plays a crucial role across ML fields including deep neural networks, reinforcement learning, meta-learning, variational inference, and MCMC, with specific methods tailored to each domain.