[Paper Review] A Versatile Multi-Agent Reinforcement Learning Benchmark for Inventory Management

Xianliang Yang, Zhihao Liu+5 more|arXiv (Cornell University)|Jun 13, 2023

Supply Chain and Inventory Management4 citations

TL;DR

This paper introduces MABIM, a versatile multi-agent reinforcement learning benchmark for multi-echelon, multi-commodity inventory management using real-world retail data. It enables evaluation of MARL algorithms across challenges like scalability, cooperation, competition, and non-stationary dynamics, revealing that hybrid approaches (e.g., IPPO+BS) outperform pure MARL or OR methods in complex settings.

ABSTRACT

Multi-agent reinforcement learning (MARL) models multiple agents that interact and learn within a shared environment. This paradigm is applicable to various industrial scenarios such as autonomous driving, quantitative trading, and inventory management. However, applying MARL to these real-world scenarios is impeded by many challenges such as scaling up, complex agent interactions, and non-stationary dynamics. To incentivize the research of MARL on these challenges, we develop MABIM (Multi-Agent Benchmark for Inventory Management) which is a multi-echelon, multi-commodity inventory management simulator that can generate versatile tasks with these different challenging properties. Based on MABIM, we evaluate the performance of classic operations research (OR) methods and popular MARL algorithms on these challenging tasks to highlight their weaknesses and potential.

Motivation & Objective

• To address the lack of comprehensive benchmarks for multi-agent reinforcement learning (MARL) in complex, real-world industrial scenarios like inventory management.
• To develop a flexible, scalable, and realistic simulation environment that captures key challenges such as agent interactions, non-stationary dynamics, and large-scale coordination.
• To evaluate both classic operations research (OR) methods and modern MARL algorithms on diverse, realistic inventory management tasks to expose algorithmic limitations.
• To provide a standardized, open-source benchmark (MABIM) that supports reproducible evaluation and drives progress in MARL for industrial applications.

Proposed method

• MABIM is built on the OpenAI Gym framework and simulates a multi-echelon, multi-commodity inventory system with real retail data from a production partner.
• The environment supports up to 2,000 agents (warehouses), dynamic context variables (e.g., demand, price, lead time), and customizable task configurations.
• Tasks are designed to test scalability, cooperation between upstream/downstream warehouses, competition among peers, and generalization/robustness under demand shifts and noise.
• The benchmark enables evaluation of MARL algorithms (e.g., IPPO, QTRAN) and OR baselines (e.g., (s,S) policy, BS) under identical, realistic conditions.
• Hybrid algorithms are introduced, such as IPPO+BS, where MARL learns upstream strategies while OR methods handle downstream decisions, improving performance.
• Performance is measured via cumulative profit, with normalization using the (s,S) hindsight baseline for fair comparison across non-stationary tasks.

Figure 1: Multi-echelon inventory model.

Experimental results

Research questions

• RQ1 How do MARL algorithms perform under large-scale agent settings (up to 2,000 agents) in a realistic inventory system?
• RQ2 What are the limitations of existing MARL algorithms in coordinating upstream and downstream warehouses in multi-echelon supply chains?
• RQ3 How well do MARL algorithms generalize to unseen demand patterns or robustly handle noisy demand fluctuations?
• RQ4 Can hybrid approaches combining MARL and OR methods outperform pure MARL or OR baselines in complex inventory tasks?
• RQ5 What are the key failure modes of MARL in inventory management, such as instability in training or poor coordination under competition?

Key findings

• In competition tasks, IPPO reduced replenishment to avoid losses but underperformed compared to (s,S) static and QTRAN, which showed better stability and profit.
• In two-echelon cooperation tasks, pure IPPO achieved only 6.72k profit, while IPPO+BS improved to 9.86k, demonstrating the benefit of combining MARL with OR policies.
• In three-echelon cooperation, IPPO dropped to 4.25k profit, but IPPO+BS reached 10.14k, indicating that MARL struggles with multi-layer coordination without external policy guidance.
• In non-stationary context tasks, IPPO outperformed other MARL algorithms in generalization and robustness, while static OR methods performed worst, especially under demand shifts.
• The (s,S) hindsight algorithm achieved the best performance and was used as a normalization baseline, showing its effectiveness in test-time optimization.
• The results highlight that MARL algorithms face significant challenges in scalability, coordination across echelons, and robustness to dynamic changes, underscoring the need for improved algorithmic designs.

Figure 2: Combined challenge. $\uparrow$ , $\downarrow$ , and ~ signify increasing, decreasing, and keep steady.