[Paper Review] A Deep Q-Network for the Beer Game with Partial Information.

Afshin Oroojlooyjadid, Mohammadreza Nazari+2 more|arXiv (Cornell University)|Aug 20, 2017

Supply Chain and Inventory Management12 references11 citations

TL;DR

This paper proposes a deep Q-network (DQN) agent to optimize replenishment decisions in the beer game—a decentralized, partially observed supply chain game—achieving near-optimal performance when paired with base-stock agents and significantly outperforming base-stock policies against human-like ordering behavior. The method enables real-time decision-making through pre-trained, transferable policies without parameter constraints.

ABSTRACT

The beer game is a widely used in-class game that is played in supply chain management classes to demonstrate the bullwhip effect. The game is a decentralized, multi-agent, cooperative problem that can be modeled as a serial supply chain network in which agents cooperatively attempt to minimize the total cost of the network even though each agent can only observe its own local information. Each agent chooses order quantities to replenish its stock. Under some conditions, a base-stock replenishment policy is known to be optimal. However, in a decentralized supply chain in which some agents (stages) may act irrationally (as they do in the beer game), there is no known optimal policy for an agent wishing to act optimally. We propose a machine learning algorithm, based on deep Q-networks, to optimize the replenishment decisions at a given stage. When playing alongside agents who follow a base-stock policy, our algorithm obtains near-optimal order quantities. It performs much better than a base-stock policy when the other agents use a more realistic model of human ordering behavior. Unlike most other algorithms in the literature, our algorithm does not have any limits on the beer game parameter values. Like any deep learning algorithm, training the algorithm can be computationally intensive, but this can be performed ahead of time; the algorithm executes in real time when the game is played. Moreover, we propose a transfer learning approach so that the training performed for one agent and one set of cost coefficients can be adapted quickly for other agents and costs. Our algorithm can be extended to other decentralized multi-agent cooperative games with partially observed information, which is a common type of situation in real-world supply chain problems.

Motivation & Objective

• To address the lack of optimal decision policies in decentralized, partially observed supply chain games like the beer game, where agents act irrationally.
• To develop a machine learning-based policy that operates effectively under partial information and without restrictions on game parameters.
• To enable real-time execution of optimized decisions after computationally intensive pre-training.
• To design a transfer learning framework allowing rapid adaptation of trained policies to new agents and cost structures.
• To extend the approach to other real-world decentralized, cooperative, partially observed multi-agent supply chain problems.

Proposed method

• A deep Q-network (DQN) is trained to learn optimal order quantities based on local state observations, using deep reinforcement learning.
• The DQN agent learns from cumulative rewards representing total network cost, optimizing for minimal cost over time.
• The algorithm is trained offline using experience replay and target networks to stabilize learning.
• A transfer learning approach reuses pre-trained DQN weights and fine-tunes them for new agents or cost coefficients, reducing training time.
• The method does not impose constraints on the beer game’s cost coefficients or parameter values.
• The trained agent operates in real time during gameplay, making decisions based on its current local state only.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn near-optimal replenishment policies in the beer game under partial information?
• RQ2 How does the DQN agent’s performance compare to a base-stock policy when other agents follow base-stock or human-like behavior?
• RQ3 To what extent can a pre-trained DQN policy be transferred to new agents or cost structures with minimal retraining?
• RQ4 Does the DQN approach remain effective across different parameter values without requiring architectural changes?
• RQ5 Can the DQN agent outperform traditional policies in decentralized, cooperative, partially observed supply chain games?

Key findings

• The DQN agent achieves near-optimal performance when other agents follow a base-stock policy, demonstrating strong learning capability.
• The DQN significantly outperforms the base-stock policy when other agents exhibit human-like, non-optimized ordering behavior.
• The transfer learning approach enables rapid adaptation of the policy to new agents and cost coefficients with minimal additional training.
• The method is effective across all parameter values of the beer game, with no restrictions on cost coefficients or system parameters.
• The DQN policy can be pre-trained and deployed in real time, enabling low-latency decision-making during gameplay.
• The approach is generalizable to other decentralized, partially observed, cooperative multi-agent supply chain problems.