[Paper Review] Value Function is All You Need: A Unified Learning Framework for Ride Hailing Platforms

Xiaocheng Tang, Fan Zhang+7 more|arXiv (Cornell University)|May 18, 2021

Transportation and Mobility Innovations18 references4 citations

TL;DR

This paper proposes V1D3, a unified value-based reinforcement learning framework for ride-hailing platforms that leverages a globally shared value function updated via online experience and offline trajectory data. By combining fast online learning with large-scale offline pretraining through a periodic ensemble method, V1D3 achieves state-of-the-art performance in both order dispatching and vehicle repositioning, outperforming KDD Cup 2020 winners on driver income and user experience metrics.

ABSTRACT

Large ride-hailing platforms, such as DiDi, Uber and Lyft, connect tens of thousands of vehicles in a city to millions of ride demands throughout the day, providing great promises for improving transportation efficiency through the tasks of order dispatching and vehicle repositioning. Existing studies, however, usually consider the two tasks in simplified settings that hardly address the complex interactions between the two, the real-time fluctuations between supply and demand, and the necessary coordinations due to the large-scale nature of the problem. In this paper we propose a unified value-based dynamic learning framework (V1D3) for tackling both tasks. At the center of the framework is a globally shared value function that is updated continuously using online experiences generated from real-time platform transactions. To improve the sample-efficiency and the robustness, we further propose a novel periodic ensemble method combining the fast online learning with a large-scale offline training scheme that leverages the abundant historical driver trajectory data. This allows the proposed framework to adapt quickly to the highly dynamic environment, to generalize robustly to recurrent patterns and to drive implicit coordinations among the population of managed vehicles. Extensive experiments based on real-world datasets show considerably improvements over other recently proposed methods on both tasks. Particularly, V1D3 outperforms the first prize winners of both dispatching and repositioning tracks in the KDD Cup 2020 RL competition, achieving state-of-the-art results on improving both total driver income and user experience related metrics.

Motivation & Objective

• To address the complex, real-time interactions between supply and demand in large-scale ride-hailing platforms.
• To unify order dispatching and vehicle repositioning into a single learning framework that captures their interdependencies.
• To improve sample efficiency and robustness in dynamic environments using both online transactions and historical driver trajectories.
• To enable fast adaptation to supply-demand shifts while maintaining long-term performance through value function learning.
• To achieve state-of-the-art results in both driver income and user experience metrics on real-world data.

Proposed method

• A globally shared value function is continuously updated using online experiences from real-time platform transactions.
• A novel periodic ensemble method combines fast online learning with large-scale offline training on historical driver trajectories.
• Online learning uses a population-based Temporal Difference objective derived from on-policy value iteration for rapid adaptation.
• The value function uses coarse-coded state representations to ensure geographically smooth value responses to supply and demand changes.
• The framework integrates value-based planning for dispatching and repositioning decisions using the same shared value function.
• The offline model provides robust initialization, while online updates allow dynamic response to real-time fluctuations.

Experimental results

Research questions

• RQ1 Can a unified value-based framework effectively coordinate order dispatching and vehicle repositioning in large-scale ride-hailing systems?
• RQ2 How can online and offline learning be combined to improve sample efficiency and robustness in dynamic transportation environments?
• RQ3 To what extent can a shared value function adapt to sudden changes in supply (drivers) and demand (orders)?
• RQ4 Does the proposed ensemble learning method outperform standalone online or offline training in real-world ride-hailing scenarios?
• RQ5 Can the framework achieve superior performance on both driver income and user experience metrics simultaneously?

Key findings

• V1D3 outperforms the first-prize solutions in both the dispatching and repositioning tracks of the KDD Cup 2020 Reinforcement Learning competition.
• The framework achieves state-of-the-art results in improving total driver income and user experience-related metrics on real-world datasets.
• The value function rapidly adapts to new driver arrivals, showing a prompt decrease in value at the source location to discourage over-recruitment.
• The value function increases in response to sudden demand surges, encouraging more drivers to be dispatched to high-demand areas.
• The dynamic value response is geographically smooth and decays naturally as supply-demand imbalances are resolved.
• The periodic ensemble method enables fast online adaptation while maintaining robustness through large-scale offline pretraining.