[论文解读] Learning to Optimize

Ke Li, Jitendra Malik|arXiv (Cornell University)|Jun 6, 2016

Machine Learning and Data Classification参考文献 26被引用 166

一句话总结

这篇论文使用引导策略搜索学习一个基于策略的优化算法，将任何优化方法视为一个策略，展示学习到的优化器在凸和非凸问题上相比手工设计的优化器能更快收敛并达到更好的最优解。

ABSTRACT

Algorithm design is a laborious process and often requires many iterations of ideation and validation. In this paper, we explore automating algorithm design and present a method to learn an optimization algorithm, which we believe to be the first method that can automatically discover a better algorithm. We approach this problem from a reinforcement learning perspective and represent any particular optimization algorithm as a policy. We learn an optimization algorithm using guided policy search and demonstrate that the resulting algorithm outperforms existing hand-engineered algorithms in terms of convergence speed and/or the final objective value.

研究动机与目标

• Motivate automating the design of unconstrained continuous optimization algorithms.
• Develop a framework where an optimization algorithm is represented as a policy in a reinforcement learning setting.
• Train a learned optimizer that converges faster and/or finds better optima than traditional algorithms.
• Demonstrate generalization of the learned optimizer to unseen objectives and longer horizons.

提出的方法

• Formulate optimization as a reinforcement learning problem where the policy determines the step to take at each iteration.
• Represent the optimizer as a policy pi that maps objective values and gradients at current and past points to a step delta x.
• Use guided policy search to learn policy parameters by alternating between constructing a target trajectory distribution and supervised learning of the policy.
• Model the policy with a small neural network (one hidden layer, 50 units, Softplus activations).
• Use a state that includes current location, changes in objective value, and past gradients over the last H=25 steps; exclude absolute coordinates.
• Train the policy using trajectories from randomly generated objective functions; initialize the target trajectory to mimic gradient-descent-with-momentum and then refine.
• Evaluate on convex (logistic regression) and non-convex (robust linear regression and a two-layer ReLU neural net) objectives to compare with hand-engineered optimizers (gradient descent, momentum, conjugate gradient, L-BFGS).

实验结果

研究问题

• RQ1 Can a learned optimization policy outperform traditional hand-engineered optimizers across different objective function classes (convex and non-convex)?
• RQ2 Does the learned optimizer generalize to unseen objective functions and longer optimization horizons than those seen during training?
• RQ3 On which problem classes does the autonomous optimizer provide the most significant improvements or show limitations compared to baselines?
• RQ4 How does the learned optimizer perform relative to state-of-the-art methods like L-BFGS on convex objectives?
• RQ5 Does the autonomous optimizer reduce oscillations and getting trapped in local optima on non-convex problems?

主要发现

• The autonomous optimizer outperforms gradient descent, momentum, and conjugate gradient on logistic regression test objectives, especially in early iterations.
• On logistic regression, L-BFGS converges slightly faster in some cases, but the autonomous optimizer remains competitive and often faster overall.
• For robust linear regression, the autonomous optimizer beats gradient descent, conjugate gradient, and L-BFGS across most iterations, with momentum sometimes catching up early.
• In neural net training, the autonomous optimizer significantly outperforms baselines, achieving faster convergence and better optima with fewer oscillations.
• Across non-convex problems (robust regression and neural nets), conjugate gradient and L-BFGS frequently diverge, while the learned optimizer maintains stability and superior performance.
• The learned optimizer generalizes to longer horizons beyond the 40-step training trajectories and can achieve comparable or better optima than baselines in test objectives.