[Paper Review] Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains

Francesco Stranieri, Fabio Stella|arXiv (Cornell University)|Apr 20, 2022

Supply Chain and Inventory Management22 references4 citations

TL;DR

This paper proposes a customizable, open-source deep reinforcement learning (DRL) environment for two-echelon supply chain inventory management under stochastic and seasonal demand, supporting arbitrary numbers of warehouses and product types. It evaluates state-of-the-art DRL algorithms—PPO, A3C, VPG, and BO—showing PPO consistently outperforms others and near-oracle performance, especially in complex topologies with multiple products and warehouses.

ABSTRACT

In this study, we analyze and compare the performance of state-of-the-art deep reinforcement learning algorithms for solving the supply chain inventory management problem. This complex sequential decision-making problem consists of determining the optimal quantity of products to be produced and shipped across different warehouses over a given time horizon. In particular, we present a mathematical formulation of a two-echelon supply chain environment with stochastic and seasonal demand, which allows managing an arbitrary number of warehouses and product types. Through a rich set of numerical experiments, we compare the performance of different deep reinforcement learning algorithms under various supply chain structures, topologies, demands, capacities, and costs. The results of the experimental plan indicate that deep reinforcement learning algorithms outperform traditional inventory management strategies, such as the static (s, Q)-policy. Furthermore, this study provides detailed insight into the design and development of an open-source software library that provides a customizable environment for solving the supply chain inventory management problem using a wide range of data-driven approaches.

Motivation & Objective

• To design a flexible, open-source DRL environment for two-echelon supply chain inventory management (SCIM) with stochastic, seasonal demand.
• To compare the performance of state-of-the-art DRL algorithms (PPO, A3C, VPG, BO) under varying supply chain topologies, product types, and operational configurations.
• To benchmark DRL performance against a data-driven (s, Q)-policy and an oracle baseline to assess effectiveness and optimality gap.
• To enable reproducible research by releasing a standardized, Gym-compatible SCIM environment under open science principles.

Proposed method

• Formalizes a two-echelon SCIM problem with finite warehouse capacities, stochastic and seasonal demand, and multiple product types using a Markov Decision Process (MDP) framework.
• Implements a custom, extensible environment using OpenAI Gym-style API to standardize state, action, and reward definitions for DRL agents.
• Applies four DRL algorithms: Proximal Policy Optimization (PPO), Advantage Actor-Critic (A3C), Value Policy Gradient (VPG), and Bayesian Optimization (BO) for hyperparameter tuning.
• Uses a profit-maximization objective with penalties for backordering and holding costs, and rewards for timely deliveries and inventory efficiency.
• Employs a data-driven approach to tune (s, Q)-policy parameters for fair comparison with DRL agents.
• Conducts extensive experiments across 15 configurations (3 scenarios × 5 experiments each), varying warehouse count, product types, demand patterns, and cost structures.

Experimental results

Research questions

• RQ1 How do different DRL algorithms (PPO, A3C, VPG, BO) perform in terms of profit maximization across diverse two-echelon supply chain topologies and configurations?
• RQ2 To what extent can DRL agents outperform traditional (s, Q)-policies in complex, stochastic, and seasonal SCIM environments?
• RQ3 How close do DRL agents come to the performance of an oracle that knows the optimal action in advance?
• RQ4 How does the scalability and robustness of DRL algorithms vary with increasing complexity (e.g., more warehouses, multiple products, higher demand variability)?

Key findings

• PPO achieved the highest average profit across all scenarios, consistently outperforming other DRL algorithms and coming closest to the oracle’s performance.
• In the 1P1W scenario, BO achieved near-oracle results (1226 ± 71 vs. 1474 ± 45), but performance degraded significantly in more complex settings with multiple products or warehouses.
• In the 1P3W scenario, PPO achieved 1600 ± 62 profit in Exp 4 (vs. oracle 2046 ± 37), while BO only reached 1633 ± 39, indicating PPO’s superior scalability.
• VPG showed stable but suboptimal convergence, often settling in local maxima, especially as the number of warehouses increased.
• A3C and VPG performed poorly in high-complexity experiments, with A3C showing negative profits in the most challenging 1P3W experiment (−2142 ± 128).
• The (s, Q)-policy, while myopic and non-adaptive, still outperformed A3C and VPG in some cases, highlighting the limitations of certain DRL approaches in complex settings.