[Paper Review] Playing hide and seek: tackling in-store picking operations while improving customer experience

Fábio Neves-Moreira, Pedro Amorim|arXiv (Cornell University)|Jan 5, 2023

Consumer Retail Behavior StudiesBusiness, Management and Accounting3 citations

TL;DR

This paper introduces the Dynamic In-store Picker Routing Problem (diPRP), modeling in-store online order picking as a Markov Decision Process (MDP) and solving it via a hybrid reinforcement learning and mathematical programming approach. The method learns policies that reduce customer encounters by over 50% compared to shortest-path strategies, balancing operational efficiency with in-store customer experience in omnichannel retail environments.

ABSTRACT

The evolution of the retail business presents new challenges and raises pivotal questions on how to reinvent stores and supply chains to meet the growing demand of the online channel. One of the recent measures adopted by omnichannel retailers is to address the growth of online sales using in-store picking, which allows serving online orders using existing assets. However, it comes with the downside of harming the offline customer experience. To achieve picking policies adapted to the dynamic customer flows of a retail store, we formalize a new problem called Dynamic In-store Picker Routing Problem (diPRP). In this relevant problem - diPRP - a picker tries to pick online orders while minimizing customer encounters. We model the problem as a Markov Decision Process (MDP) and solve it using a hybrid solution approach comprising mathematical programming and reinforcement learning components. Computational experiments on synthetic instances suggest that the algorithm converges to efficient policies. Furthermore, we apply our approach in the context of a large European retailer to assess the results of the proposed policies regarding the number of orders picked and customers encountered. Our work suggests that retailers should be able to scale the in-store picking of online orders without jeopardizing the experience of offline customers. The policies learned using the proposed solution approach reduced the number of customer encounters by more than 50% when compared to policies solely focused on picking orders. Thus, to pursue omnichannel strategies that adequately trade-off operational efficiency and customer experience, retailers cannot rely on actual simplistic picking strategies, such as choosing the shortest possible route.

Motivation & Objective

• To address the growing challenge of in-store online order picking in omnichannel retail while minimizing disruption to in-store customers.
• To formalize a new dynamic routing problem—diPRP—that accounts for real-time customer flow and picker efficiency.
• To develop a scalable, low-computation policy that balances order picking performance and customer experience.
• To validate the approach using both synthetic instances and a real-world case from a large European retailer.
• To demonstrate that simplistic shortest-path policies significantly increase customer encounters, undermining in-store experience.

Proposed method

• Formalizes the diPRP as a Markov Decision Process (MDP) with states defined by picker position and target location.
• Integrates mathematical programming to generate initial routing policies and simulate customer flows in a dynamic store environment.
• Employs Q-learning (QL) with state-action value function approximation to train picker agents to avoid crowded nodes.
• Uses arc-based distances and crowd-aware features as state variables to guide policy learning.
• Combines simulation-based training with reinforcement learning to learn policies that minimize customer encounters.
• Applies a hybrid solution framework: simulation for environment dynamics, RL for policy learning, and mathematical programming for route initialization.

Experimental results

Research questions

• RQ1 How can in-store picker routing be optimized to reduce customer encounters while maintaining high order-picking efficiency?
• RQ2 To what extent can reinforcement learning learn effective policies in a dynamic, real-time retail environment with unpredictable customer flows?
• RQ3 How does a crowd-aware routing policy compare to shortest-path-based policies in terms of customer disturbance and order throughput?
• RQ4 Can a hybrid approach combining simulation, mathematical programming, and RL achieve scalable, low-computation decision-making for in-store pickers?
• RQ5 What managerial insights can be derived from real-world application of such a policy in a large European retailer?

Key findings

• The proposed reinforcement learning policy reduced customer encounters by more than 50% compared to shortest-path-based policies in both synthetic and real-world instances.
• The QL policy, trained on arc distances and crowd-aware features, learned to avoid high-traffic nodes despite taking longer paths, significantly reducing customer interactions.
• The shortest path (SP) policy, while efficient in distance, visited the most crowded nodes, resulting in nearly double the number of customer encounters.
• The CN policy, which prioritized avoiding customers at all costs, took the longest path, demonstrating that optimal trade-offs require balancing efficiency and avoidance.
• The hybrid solution approach achieved effective policy learning in a reasonable number of training episodes with low computational overhead and fast inference times.
• Real-world application confirmed that crowd-aware routing significantly improves in-store customer experience without sacrificing order-picking performance.