[Paper Review] Optimizing Taxi Carpool Policies via Reinforcement Learning and Spatio-Temporal Mining

Ishan Jindal, Zhiwei Qin+3 more|arXiv (Cornell University)|Nov 11, 2018

Transportation and Mobility Innovations23 references4 citations

TL;DR

This paper proposes a reinforcement learning (RL) framework combined with a spatio-temporal neural network (ST-NN) to optimize taxi carpooling policies for improved transportation efficiency. The ST-NN predicts travel time and distance directly from raw GPS data without feature engineering, while the RL agent learns when to accept carpool requests to maximize effective trip distance. The method outperforms fixed and tabular Q policies, especially in low-demand areas like Uptown Manhattan, demonstrating superior long-term efficiency and reduced congestion.

ABSTRACT

In this paper, we develop a reinforcement learning (RL) based system to learn an effective policy for carpooling that maximizes transportation efficiency so that fewer cars are required to fulfill the given amount of trip demand. For this purpose, first, we develop a deep neural network model, called ST-NN (Spatio-Temporal Neural Network), to predict taxi trip time from the raw GPS trip data. Secondly, we develop a carpooling simulation environment for RL training, with the output of ST-NN and using the NYC taxi trip dataset. In order to maximize transportation efficiency and minimize traffic congestion, we choose the effective distance covered by the driver on a carpool trip as the reward. Therefore, the more effective distance a driver achieves over a trip (i.e. to satisfy more trip demand) the higher the efficiency and the less will be the traffic congestion. We compared the performance of RL learned policy to a fixed policy (which always accepts carpool) as a baseline and obtained promising results that are interpretable and demonstrate the advantage of our RL approach. We also compare the performance of ST-NN to that of state-of-the-art travel time estimation methods and observe that ST-NN significantly improves the prediction performance and is more robust to outliers.

Motivation & Objective

• To develop a data-driven carpooling policy that maximizes transportation efficiency and minimizes traffic congestion in urban taxi systems.
• To address the challenge of accurate travel time and distance estimation in dynamic urban environments using raw GPS data.
• To design a reinforcement learning framework that learns optimal carpool acceptance decisions based on real-time driver state and future demand prospects.
• To evaluate the performance of the RL policy against fixed and tabular Q policies in diverse urban regions with varying taxi demand densities.
• To demonstrate the robustness and scalability of the ST-NN model in predicting travel times and distances without manual feature engineering.

Proposed method

• Developed a spatio-temporal neural network (ST-NN) that takes raw GPS coordinates of origin and destination and time-of-day as input to predict travel time and distance, eliminating the need for route or map construction.
• Built a carpooling simulation environment using NYC taxi trip data, where the ST-NN provides travel time estimates and the RL agent selects actions based on state transitions.
• Formulated the carpooling decision problem as a Markov Decision Process (MDP), with the reward defined as the effective distance covered per trip to incentivize higher efficiency and reduced congestion.
• Trained a Deep Q-Network (DQN) agent to learn a policy that selects whether to accept a carpool request at each decision point, based on maximizing long-term cumulative reward.
• Used mini-batch experience replay and target networks in DQN training to stabilize learning and improve convergence.
• Compared the DQN policy against a fixed policy (always accept) and a tabular Q-learning policy to evaluate performance across different demand regimes in Manhattan.

Experimental results

Research questions

• RQ1 Can a deep learning model trained on raw GPS data accurately predict travel time and distance without requiring map or route information?
• RQ2 Does a reinforcement learning agent trained on simulated taxi trip data learn a superior carpooling policy compared to a fixed policy in terms of effective trip distance and congestion reduction?
• RQ3 How does the performance of the RL policy vary across regions with different taxi demand densities, such as Uptown versus Downtown Manhattan?
• RQ4 To what extent does the ST-NN model improve prediction accuracy and robustness to outliers compared to state-of-the-art travel time estimation methods?
• RQ5 Can the RL agent learn to trade off short-term rewards for long-term gains by strategically moving toward high-value regions, especially in low-demand areas?

Key findings

• The DQN-learned policy outperformed the fixed policy and tabular Q policy in Uptown Manhattan, achieving a mean cumulative reward of 46.08 on weekdays and 27.86 on weekends, compared to 41.543 and 25.39 for the fixed policy.
• In Downtown Manhattan, where taxi demand is high, the DQN policy performed nearly identically to the fixed policy (339.42 vs. 340.06 on weekdays), indicating that always accepting carpool is near-optimal in dense regions.
• The ST-NN model significantly outperformed state-of-the-art travel time estimation methods in prediction accuracy and robustness to outliers, without requiring manual feature engineering.
• The DQN agent learned to selectively accept carpool requests, especially in low-demand areas like Uptown, by prioritizing movement toward regions with higher long-term Q-values, even at the cost of immediate rewards.
• The tabular Q policy performed worst due to the infeasibility of maintaining Q-values across the vast state-action space, leading to sparse Q-value tables and frequent zero-value predictions at test time.
• In Uptown Manhattan, the DQN policy diverged from the fixed policy in the afternoon, strategically delaying immediate pickups to access higher-value future states, demonstrating intelligent long-term planning.