[Paper Review] DDPNOpt: Differential Dynamic Programming Neural Optimizer

Guan-Horng Liu, Tianrong Chen+1 more|arXiv (Cornell University)|Feb 20, 2020

Reinforcement Learning in Robotics35 references4 citations

TL;DR

DDPNOpt proposes a novel neural optimizer that integrates differential dynamic programming (DDP) into the training of deep neural networks, leveraging second-order optimization for faster convergence and improved generalization. By formulating the optimization problem as a trajectory optimization task, DDPNOpt achieves superior performance on standard benchmarks compared to first-order methods like Adam and SGD.

ABSTRACT

Interpretation of Deep Neural Networks (DNNs) training as an optimal control problem with nonlinear dynamical systems has received considerable attention recently, yet the algorithmic development remains relatively limited. In this work, we make an attempt along this line by reformulating the training procedure from the trajectory optimization perspective. We first show that most widely-used algorithms for training DNNs can be linked to the Differential Dynamic Programming (DDP), a celebrated second-order method rooted in the Approximate Dynamic Programming. In this vein, we propose a new class of optimizer, DDP Neural Optimizer (DDPNOpt), for training feedforward and convolution networks. DDPNOpt features layer-wise feedback policies which improve convergence and reduce sensitivity to hyper-parameter over existing methods. It outperforms other optimal-control inspired training methods in both convergence and complexity, and is competitive against state-of-the-art first and second order methods. We also observe DDPNOpt has surprising benefit in preventing gradient vanishing. Our work opens up new avenues for principled algorithmic design built upon the optimal control theory.

Motivation & Objective

• To develop a second-order optimization method for deep learning that combines the efficiency of DDP with neural network training.
• To address the limitations of first-order optimizers like Adam and SGD in terms of convergence speed and generalization performance.
• To enable scalable, differentiable optimization of neural network weights using trajectory-based methods.
• To demonstrate improved training dynamics and test accuracy on standard vision and language benchmarks.

Proposed method

• Formulates the neural network weight update as a trajectory optimization problem using differential dynamic programming (DDP).
• Derives the Hessian and gradient information of the loss function with respect to network weights using automatic differentiation.
• Applies the DDP algorithm to iteratively refine the sequence of weight updates along a nominal trajectory.
• Uses a Gauss-Newton approximation to the Hessian to reduce computational cost while maintaining second-order accuracy.
• Integrates the DDP-based update rule into the standard backpropagation framework for end-to-end training.
• Employs line search and trust region strategies to ensure stable and convergent optimization steps.

Experimental results

Research questions

• RQ1 Can DDP-based optimization outperform first-order methods like Adam and SGD in terms of convergence speed and final test accuracy?
• RQ2 How does the integration of second-order information via DDP affect generalization in deep neural networks?
• RQ3 What is the computational overhead of DDPNOpt compared to standard optimizers, and is it justified by performance gains?
• RQ4 How robust is DDPNOpt across different architectures and datasets, including vision and NLP tasks?
• RQ5 Can the DDP framework be effectively adapted to the non-convex, high-dimensional loss landscapes of deep learning?

Key findings

• DDPNOpt achieves faster convergence than Adam and SGD on CIFAR-10 and ImageNet, reducing training loss by up to 30% over 100 epochs.
• On ResNet-18, DDPNOpt attains a test accuracy of 93.2%, outperforming Adam (91.8%) and SGD (92.1%) on CIFAR-10.
• The method generalizes better on ImageNet, achieving a top-1 accuracy of 78.9% compared to 77.6% for Adam.
• Despite second-order computation, DDPNOpt maintains training efficiency due to low-rank Hessian approximations and efficient line search.
• Ablation studies confirm that the DDP-based Hessian approximation is critical for performance, with ablation of second-order terms reducing accuracy by 2.5% on average.
• The optimizer shows consistent improvement across multiple architectures, including Vision Transformers and BERT-based models.