[Paper Review] Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications

Achref Bachouch, Côme Huré+2 more|arXiv (Cornell University)|Dec 13, 2018

Stochastic processes and financial applicationsEconomics, Econometrics and Finance25 references68 citations

TL;DR

This paper proposes deep learning-based algorithms—NNcontPI, Hybrid-Now, Hybrid-LaterQ, and ClassifHybrid—for solving high-dimensional finite-horizon stochastic control problems using neural networks to approximate value functions and optimal policies via performance and hybrid iteration. The methods achieve competitive or superior performance compared to quantization-based benchmarks like Qknn, especially in high dimensions, with empirical validation on 100D PDEs, option hedging, and microgrid management problems.

ABSTRACT

This paper presents several numerical applications of deep learning-based algorithms that have been introduced in [HPBL18]. Numerical and comparative tests using TensorFlow illustrate the performance of our different algorithms, namely control learning by performance iteration (algorithms NNcontPI and ClassifPI), control learning by hybrid iteration (algorithms Hybrid-Now and Hybrid-LaterQ), on the 100-dimensional nonlinear PDEs examples from [EHJ17] and on quadratic backward stochastic differential equations as in [CR16]. We also performed tests on low-dimension control problems such as an option hedging problem in finance, as well as energy storage problems arising in the valuation of gas storage and in microgrid management. Numerical results and comparisons to quantization-type algorithms Qknn, as an efficient algorithm to numerically solve low-dimensional control problems, are also provided; and some corresponding codes are available on https://github.com/comeh/.

Motivation & Objective

• To address the curse of dimensionality in finite-horizon stochastic control problems using deep neural networks.
• To extend actor-critic reinforcement learning methods to finite-horizon problems by introducing performance and hybrid iteration algorithms.
• To provide numerical validation of deep learning-based algorithms on high-dimensional and low-dimensional control problems.
• To compare the proposed algorithms against established quantization-based methods (Qknn) in terms of accuracy and computational efficiency.
• To demonstrate the feasibility and effectiveness of deep learning in solving complex, real-world control problems such as gas storage and microgrid management.

Proposed method

• Uses deep neural networks (DNNs) to parameterize the control policy and value function, with parameters θ and β respectively.
• Employs performance iteration (PI) and hybrid iteration schemes to iteratively improve policy and value function estimates using Monte Carlo regression.
• Applies two training strategy variants: exploitation-only (knowledge-based) and explore-then-exploit (using empirical measures from initial control estimates).
• Implements a hybrid approach (ClassifHybrid) combining classification and regression for mixed discrete-continuous control spaces, such as in microgrid management.
• Uses training measures µn designed via uniform or empirical distributions to sample states for network training at each time step.
• Pre-trains value function networks using weights from the next time step to accelerate convergence and improve temporal stability.

Experimental results

Research questions

• RQ1 Can deep learning-based algorithms effectively solve high-dimensional stochastic control problems where traditional methods fail due to the curse of dimensionality?
• RQ2 How do performance iteration and hybrid iteration algorithms compare in terms of accuracy and computational cost on 100-dimensional nonlinear PDEs and quadratic BSDEs?
• RQ3 To what extent do the proposed deep learning methods outperform quantization-based algorithms like Qknn in low-dimensional control problems?
• RQ4 How does the choice of training distribution (exploitation vs. exploration) impact the convergence and accuracy of the learned policies?
• RQ5 Can hybrid neural network architectures handle mixed discrete-continuous control spaces, such as in microgrid management, more effectively than standard methods?

Key findings

• ClassifHybrid achieved a value function estimate of 33.34 (±0.31) for the microgrid problem at N=30, outperforming Qknn’s estimate of 35.37 (±0.34).
• For N=200, Qknn estimated the value function at 231.8 (±1.2), indicating stable performance over long horizons.
• Hybrid-Now outperformed Qknn in the microgrid problem, though it required 7 minutes compared to Qknn’s sub-minute runtime.
• The estimated optimal decisions from Qknn showed natural behavior: turning the generator on when battery capacity is insufficient and off when demand is negative or battery is full.
• ClassifHybrid successfully managed the mixed discrete-continuous control space in microgrid management, achieving better results than Qknn despite the complexity.
• Pre-training value function networks using weights from the next time step significantly reduced training time and improved temporal stability of estimates.