[Paper Review] Solving the Order Batching and Sequencing Problem using Deep Reinforcement Learning

Bram Cals, Yingqian Zhang+2 more|arXiv (Cornell University)|Jun 16, 2020

Advanced Manufacturing and Logistics Optimization28 references4 citations

TL;DR

This paper proposes a Deep Reinforcement Learning (DRL) approach using Proximal Policy Optimization (PPO) to solve the Order Batching and Sequencing Problem (OBSP) in hybrid Person-to-Goods (PtG) and Goods-to-Person (GtP) warehouses, formulating it as a Semi-Markov Decision Process with a vector-based state representation. The DRL agent outperforms traditional heuristics, achieving lower tardy order rates and demonstrating strong generalization across varying order volumes and time periods.

ABSTRACT

In e-commerce markets, on time delivery is of great importance to customer satisfaction. In this paper, we present a Deep Reinforcement Learning (DRL) approach for deciding how and when orders should be batched and picked in a warehouse to minimize the number of tardy orders. In particular, the technique facilitates making decisions on whether an order should be picked individually (pick-by-order) or picked in a batch with other orders (pick-by-batch), and if so with which other orders. We approach the problem by formulating it as a semi-Markov decision process and develop a vector-based state representation that includes the characteristics of the warehouse system. This allows us to create a deep reinforcement learning solution that learns a strategy by interacting with the environment and solve the problem with a proximal policy optimization algorithm. We evaluate the performance of the proposed DRL approach by comparing it with several batching and sequencing heuristics in different problem settings. The results show that the DRL approach is able to develop a strategy that produces consistent, good solutions and performs better than the proposed heuristics.

Motivation & Objective

• To address the challenge of minimizing tardy orders in e-commerce warehouses under dynamic order arrivals and mixed PtG/GtP systems.
• To develop a learning-based strategy that dynamically decides between pick-by-order and pick-by-batch, and sequences batches effectively.
• To evaluate the performance and generalization capability of a DRL agent compared to conventional heuristics in complex, real-world warehouse scenarios.
• To demonstrate the feasibility and robustness of DRL in solving NP-hard sequential decision-making problems in warehousing.

Proposed method

• The OBSP is formulated as a Semi-Markov Decision Process (SMDP), enabling delayed rewards and variable action durations.
• A vector-based state representation encodes warehouse characteristics, including order due times, item locations, and system type (PtG or GtP).
• A Deep Q-Network (DQN) with a policy gradient approach is trained using the Proximal Policy Optimization (PPO) algorithm to learn optimal batching and sequencing policies.
• The agent interacts with a simulated warehouse environment that computes the impact of each action, including tardiness and walking distances.
• The DRL model is trained on diverse problem instances and evaluated for performance and generalization across different order volumes and time windows.
• Generalization is tested by applying agents trained on specific instance sizes or time periods to unseen, out-of-distribution settings.

Experimental results

Research questions

• RQ1 Can a DRL agent learn a superior batching and sequencing strategy compared to traditional heuristics in a hybrid PtG/GtP warehouse system?
• RQ2 How well does the DRL agent generalize to problem instances with different numbers of orders than seen during training?
• RQ3 Does the DRL agent maintain consistent performance when applied to different time periods, including peak hours with high order variability?
• RQ4 Can the DRL approach effectively handle dynamic order arrivals and uncertainties in e-commerce order picking environments?

Key findings

• The DRL agent achieved an average of 0.75% tardy orders on 500-order instances, outperforming all tested heuristics.
• The agent trained on 400 orders produced 1.08% tardy orders on 400-order test instances and 3.57% on 500-order instances, showing strong generalization to larger volumes.
• The agent trained on the 19:00–20:00 window achieved 0.66% average tardy orders when tested on random hours between 19:00 and 00:00, indicating robustness to temporal variation.
• When the last hour (23:00–00:00) was included in testing, performance degraded slightly, with a 0.94% average tardy rate, due to rare, small-instance states not seen during training.
• The DRL model showed consistent performance across diverse settings, with low standard deviations (e.g., 1.68% and 2.59% for 400 and 500 orders), indicating stability.
• The results confirm that DRL can learn effective, adaptive strategies for complex, dynamic warehouse decision-making problems where traditional heuristics struggle.