[Paper Review] Robust On-Line ADP-based Solution of a Class of Hierarchical Nonlinear Differential Game

Mohammad reza Satouri, Hamed Kebriaei+2 more|arXiv (Cornell University)|Jul 26, 2019

Adaptive Dynamic Programming Control4 citations

TL;DR

This paper proposes a novel online adaptive dynamic programming (ADP) method for solving hierarchical nonlinear differential games involving one leader and multiple followers under model uncertainty and disturbances. By integrating neural networks to estimate value functions, control policies, and disturbances, the algorithm reduces neural network usage by ~30% and achieves robust convergence via Lyapunov-based analysis, demonstrating effectiveness in handling both zero-sum and nonzero-sum game components under worst-case disturbances.

ABSTRACT

In this paper, a hierarchical one-leader-multi-followers game for a class of continuous-time nonlinear systems with disturbance is investigated by a novel policy iteration reinforcement learning technique in which, the game model consists both of the zero-sum and nonzero-sum games, simultaneously. An adaptive dynamic programming (ADP), method is developed to achieve optimal control strategy under the worst case of disturbance. This algorithm reduces the number of neural networks which are used for estimation for about thirty percent. The proposed algorithm uses neural networks to estimate value functions, control policies and disturbances. Convergence analysis of the estimations is investigated using Lyapunov theory and exploiting properties of the Nemytskii operator. Finally, the simulation results will show effectiveness of the developed ADP method.

Motivation & Objective

• To address the challenge of solving hierarchical one-leader-multi-followers nonlinear differential games under model uncertainty and worst-case disturbances.
• To develop an efficient online ADP-based solution that reduces computational complexity by minimizing neural network usage.
• To ensure robustness against disturbances and convergence to the Stackelberg-Nash equilibrium using Lyapunov stability theory.
• To integrate neural networks for joint estimation of value functions, control policies, and disturbance signals in a unified framework.
• To validate the method through simulation, demonstrating its effectiveness in handling both zero-sum and nonzero-sum game components simultaneously.

Proposed method

• Employs a policy iteration-based ADP framework to iteratively update control policies and value function approximations in real time.
• Uses neural networks to estimate the value function, control policy, and disturbance signal, reducing the number of required networks by approximately 30%.
• Applies Lyapunov theory and properties of the Nemytskii operator to prove uniform ultimate boundedness (UUB) of estimation errors and system states.
• Introduces a modified cost function that combines zero-sum and nonzero-sum game components, enabling simultaneous optimization.
• Utilizes gradient descent for online weight updates of neural networks, ensuring real-time adaptation.
• Derives sufficient conditions for stability by bounding the error dynamics using matrix inequalities and tuning parameters $F_1^i$, $F_2^i$ to ensure positive definiteness of the Hessian matrix $M$.

Experimental results

Research questions

• RQ1 How can an online ADP method be designed to solve hierarchical nonlinear differential games with both zero-sum and nonzero-sum components?
• RQ2 What is the minimal number of neural networks required to achieve robust control under model uncertainty and disturbances?
• RQ3 How can Lyapunov stability theory be applied to prove convergence of the ADP-based estimator in the presence of disturbances?
• RQ4 What conditions ensure the uniform ultimate boundedness (UUB) of the estimation errors and system states in the proposed framework?
• RQ5 How does the proposed method compare in performance and complexity to existing ADP-based solutions for dynamic games?

Key findings

• The proposed ADP method reduces the number of neural networks used for estimation by approximately 30% compared to conventional approaches.
• The estimation errors and system states are proven to be uniformly ultimately bounded (UUB), ensuring long-term stability under worst-case disturbances.
• The convergence of the algorithm to the Stackelberg-Nash equilibrium is analytically established using Lyapunov theory and Nemytskii operator properties.
• The method effectively handles both zero-sum and nonzero-sum game components within a single unified framework, enabling robust control under uncertainty.
• Simulation results confirm the effectiveness of the algorithm in achieving optimal control strategies despite model uncertainty and disturbances.
• Tuning parameters $F_1^i$ and $F_2^i$ are shown to be critical in ensuring the positive definiteness of matrix $M$, which governs the stability region and convergence speed.