[Paper Review] Reinforcement Learning for Multi-Product Multi-Node Inventory Management in Supply Chains

Nazneen N. Sultana, Hardik Meisheri+4 more|arXiv (Cornell University)|Jun 7, 2020

Supply Chain and Inventory ManagementBusiness, Management and Accounting35 references17 citations

TL;DR

This paper proposes a hierarchical multi-agent reinforcement learning (MARL) framework using Advantage Actor-Critic (A2C) with quantized action spaces to solve large-scale, multi-product, multi-node inventory management in supply chains. It demonstrates that the RL approach outperforms heuristic methods and approaches clairvoyant performance, even handling dynamic changes like new products and stores via transfer learning.

ABSTRACT

This paper describes the application of reinforcement learning (RL) to multi-product inventory management in supply chains. The problem description and solution are both adapted from a real-world business solution. The novelty of this problem with respect to supply chain literature is (i) we consider concurrent inventory management of a large number (50 to 1000) of products with shared capacity, (ii) we consider a multi-node supply chain consisting of a warehouse which supplies three stores, (iii) the warehouse, stores, and transportation from warehouse to stores have finite capacities, (iv) warehouse and store replenishment happen at different time scales and with realistic time lags, and (v) demand for products at the stores is stochastic. We describe a novel formulation in a multi-agent (hierarchical) reinforcement learning framework that can be used for parallelised decision-making, and use the advantage actor critic (A2C) algorithm with quantised action spaces to solve the problem. Experiments show that the proposed approach is able to handle a multi-objective reward comprised of maximising product sales and minimising wastage of perishable products.

Motivation & Objective

• Address the challenge of concurrent inventory management for 50 to 1000 products with shared capacity across a warehouse and multiple stores.
• Model a real-world two-echelon supply chain with finite capacities in warehouse, stores, and transportation, and with stochastic, time-lagged replenishment.
• Develop a scalable, parallelizable decision-making system using multi-agent RL that coordinates across different time scales and nodes.
• Enable transfer learning for dynamic scalability—handling new products and new stores without full retraining.
• Optimize a multi-objective reward balancing sales maximization and perishable product wastage minimization.

Proposed method

• Formulate the supply chain as a hierarchical multi-agent reinforcement learning problem, with stores and warehouse as independent agents.
• Use Advantage Actor-Critic (A2C) algorithm with quantized action spaces to handle continuous replenishment decisions in discrete action steps.
• Decompose training into two stages: first train store agents assuming infinite warehouse capacity, then train the warehouse agent using cumulative store performance as part of its scalar reward.
• Incorporate time-lagged replenishment and finite capacities (warehouse, stores, trucks) into the environment dynamics.
• Apply transfer learning by reusing pre-trained models: fine-tune normalization statistics for new products or new stores without retraining.
• Define a multi-objective reward function combining sales maximization and wastage minimization for perishable goods.

Experimental results

Research questions

• RQ1 Can a hierarchical multi-agent RL framework effectively coordinate replenishment decisions across multiple products and nodes in a finite-capacity supply chain?
• RQ2 How well does the proposed RL approach perform compared to heuristic policies and a clairvoyant oracle in terms of sales and wastage?
• RQ3 To what extent can transfer learning enable the RL model to generalize to new products and new stores without retraining?
• RQ4 How do the learned policies respond to changes in inventory levels and demand forecasts across different product counts (50, 220, 1000)?
• RQ5 Can the RL system maintain performance under system constraints such as shared truck capacity and stochastic demand?

Key findings

• The RL agent outperformed the heuristic policy in all 50, 220, and 1000 product scenarios, with performance gains of up to 46% in warehouse and 40% in stores over the heuristic.
• For 50 products, the RL model achieved a normalized performance of 0.428 (warehouse) and 0.764–0.796 (stores), surpassing the heuristic (0.299–0.673) and approaching the clairvoyant policy (0.498–0.837).
• Transfer learning enabled the warehouse model to handle an additional store (Store 4) with 1.75x capacity of Store 1 without retraining, achieving 0.446 performance—better than the heuristic (0.373) and close to clairvoyant (0.544).
• When scaling to 1000 products, the RL policy requested lower average replenishment due to capacity constraints, reflecting a realistic response to the law of large numbers.
• Policy heatmaps revealed that replenishment actions increase with low inventory and high forecast, and decrease as inventory rises or forecast drops, confirming intuitive and stable behavior.
• The model demonstrated robustness to dynamic changes: it seamlessly adapted to new products and new stores through normalization adaptation, without architectural or training changes.