[Paper Review] Gamifying the Vehicle Routing Problem with Stochastic Requests

Nicholas Kullman, Nikita Dudorov+3 more|arXiv (Cornell University)|Nov 14, 2019

Optimization and Search Problems15 references4 citations

TL;DR

This paper introduces 'Atari-fication'—a novel method to reformulate complex sequential decision-making problems, such as the vehicle routing problem with stochastic service requests (VRPSSR), as Atari-like video games to leverage deep reinforcement learning (DRL) agents. The approach maps problem states to visual pixel representations, enabling DRL agents to learn optimal routing policies through trial-and-error learning, with preliminary results showing promise for solving stochastic VRPs using DRL frameworks.

ABSTRACT

Do you remember your first video game console? We remember ours. Decades ago, they provided hours of entertainment. Now, we have repurposed them to solve dynamic and stochastic optimization problems. With deep reinforcement learning methods posting superhuman performance on a wide range of Atari games, we consider the task of representing a classic logistics problem as a game. Then, we train agents to play it. We consider several game designs for the vehicle routing problem with stochastic requests. We show how various design features impact agents' performance, including perspective, field of view, and minimaps. With the right game design, general purpose Atari agents outperform optimization-based benchmarks, especially as problem size grows. Our work points to the representation of dynamic and stochastic optimization problems via games as a promising research direction.

Motivation & Objective

• To develop a generalizable method for modeling complex operations research problems as Atari-like video games to enable application of state-of-the-art deep reinforcement learning (DRL) techniques.
• To demonstrate the feasibility of applying DRL agents—originally trained on Atari games—to real-world sequential decision-making problems involving uncertainty, such as the VRPSSR.
• To explore whether visual state representations can effectively encode complex problem structures, enabling DRL agents to learn meaningful policies without explicit feature engineering.
• To evaluate the potential of Atari-fication as a scalable, reusable framework for solving a broad class of NP-hard problems involving dynamic uncertainty and sequential decisions.
• To identify the limitations and constraints of the approach, particularly in multi-agent settings and when preserving structural properties like distance matrices.

Proposed method

• The problem is reformulated as a grid-based, two-dimensional, third-person video game where the environment state is represented as a pixel grid, with each cell encoding relevant problem features such as vehicle location, customer demand, and time window status.
• State representations use discrete pixel values: 0 for inactive or out-of-window customers, 10 for open windows with >15 minutes remaining, and 20 otherwise, enabling visual input to DRL agents.
• The DRL agent uses a deep Q-network (DQN) architecture to learn Q-values for state-action pairs, selecting actions (e.g., move to next customer) that maximize cumulative expected reward.
• The environment is designed to mimic Atari games in format—2D, screen-based, with discrete actions and sparse rewards—making it compatible with existing DRL training pipelines like OpenAI Baselines.
• The method leverages the visual perception capability of DRL agents, allowing them to process raw pixel inputs without requiring handcrafted state features, similar to how humans interpret visual scenes.
• The approach is generalizable: any problem with a sequential decision structure and visualizable state space can be Atari-fied, including TSP, knapsack, and scheduling problems.

Experimental results

Research questions

• RQ1 Can a vehicle routing problem with stochastic service requests be effectively reformulated as an Atari-like video game to enable training of deep reinforcement learning agents?
• RQ2 To what extent can existing DRL agents, pre-trained on Atari games, generalize to solve a new, complex operations research problem like the VRPSSR?
• RQ3 How well do visual state representations preserve the essential structural and dynamic properties of the original problem, such as time windows and vehicle capacities?
• RQ4 What are the limitations of Atari-fication when applied to multi-agent or constraint-sensitive problems, such as multi-vehicle routing or problems requiring exact distance matrices?
• RQ5 Can Atari-fication serve as a reusable framework for solving a wide range of sequential decision-making problems under uncertainty?

Key findings

• The Atari-fication approach successfully maps the VRPSSR into a game-like environment with visual state representation, enabling DRL agents to learn routing policies through reinforcement learning.
• Preliminary results show that DRL agents trained on the Atari-fied VRPSSR environment achieve competitive performance, demonstrating the feasibility of the method despite not being transformative.
• The method preserves key problem features such as time windows and stochastic customer requests through pixel-based state encoding, enabling the agent to reason about dynamic uncertainty.
• The approach is generalizable: similar visual reformulations could be applied to other NP-hard problems like TSP, knapsack, and scheduling problems, potentially enabling DRL to solve them via game-like interfaces.
• Limitations include potential loss of exact distance matrix fidelity and challenges in multi-agent settings, where action mapping to visual entities becomes ambiguous.
• The framework enables reuse and sharing of Atari-fied problem formulations, reducing setup time for future research and promoting standardization in DRL-based operations research.