[Paper Review] Two Phase $Q-$learning for Bidding-based Vehicle Sharing

Yinlam Chow, Jia Yuan Yu+1 more|arXiv (Cornell University)|Sep 29, 2015

Transportation and Mobility Innovations25 references3 citations

TL;DR

This paper proposes a two-phase Q-learning algorithm for bidding-based vehicle sharing systems, where customers submit bids and the operator optimally assigns rentals to maximize revenue while enforcing a vehicle utilization constraint. The method models the problem as a constrained Markov decision process (CMDP), derives a two-phase Bellman optimality condition, and demonstrates through numerical experiments that the algorithm converges faster and better satisfies service constraints than alternatives like penalized Q-learning or Lagrangian actor-critic methods.

ABSTRACT

We consider one-way vehicle sharing systems where customers can rent a car at one station and drop it off at another. The problem we address is to optimize the distribution of cars, and quality of service, by pricing rentals appropriately. We propose a bidding approach that is inspired from auctions and takes into account the significant uncertainty inherent in the problem data (e.g., pick-up and drop-off locations, time of requests, and duration of trips). Specifically, in contrast to current vehicle sharing systems, the operator does not set prices. Instead, customers submit bids and the operator decides whether to rent or not. The operator can even accept negative bids to motivate drivers to rebalance available cars to unpopular destinations within a city. We model the operator's sequential decision-making problem as a \emph{constrained Markov decision problem} (CMDP) and propose and rigorously analyze a novel two phase $Q$-learning algorithm for its solution. Numerical experiments are presented and discussed.

Motivation & Objective

• To address demand-supply imbalances in one-way vehicle sharing systems by replacing fixed pricing with an auction-style bidding mechanism.
• To model the operator's sequential decision-making as a constrained Markov decision process (CMDP) with revenue maximization and vehicle utilization constraints.
• To develop a sampling-based two-phase Q-learning algorithm that converges to an optimal policy under mild assumptions.
• To empirically evaluate the algorithm’s performance against existing methods, including penalized Q-learning and Lagrangian actor-critic approaches.
• To provide a theoretically grounded, scalable solution that reduces reliance on crew drivers and parking space overcapacity.

Proposed method

• The operator uses a bidding mechanism where customers submit bids, and the operator decides which bids to accept, enabling dynamic pricing and rebalancing incentives.
• The problem is formulated as a CMDP where actions correspond to rental assignment decisions based on bids, with the objective of maximizing total revenue subject to a minimum average vehicle utilization constraint.
• A two-phase Bellman optimality condition is derived, showing that CMDPs can be solved via two-phase dynamic programming: first solving for the value function under a given Lagrange multiplier, then optimizing the multiplier.
• The proposed two-phase Q-learning algorithm alternates between updating Q-functions for rental decisions and adjusting the Lagrange multiplier to enforce the utilization constraint.
• In each iteration, the assignment subproblem is cast as a bilinear integer linear program (BILP), solvable with standard solvers like CPLEX for medium-scale problems.
• The algorithm is designed to converge asymptotically to the optimal solution of the CMDP under mild regularity conditions, with faster convergence than actor-critic-based Lagrangian methods.

Experimental results

Research questions

• RQ1 Can a bidding mechanism effectively replace fixed pricing in one-way vehicle sharing to improve system efficiency and reduce operational costs?
• RQ2 How can the operator’s sequential decision-making under revenue and utilization constraints be optimally modeled as a constrained Markov decision process (CMDP)?
• RQ3 What is the theoretical structure of the optimal policy for CMDPs in this context, and can it be decomposed into a two-phase dynamic programming procedure?
• RQ4 How does the proposed two-phase Q-learning algorithm compare to existing methods like penalized Q-learning and Lagrangian actor-critic in terms of convergence speed and constraint satisfaction?
• RQ5 Can the algorithm scale to medium-sized vehicle sharing systems while maintaining performance guarantees?

Key findings

• The two-phase Q-learning algorithm achieved a 20% reduction in total revenue compared to unconstrained Q-learning, but ensured an average vehicle utilization time of 2.5 hours, satisfying the minimum utilization constraint.
• The algorithm converged faster than the Lagrangian actor-critic method, which requires sequential gradient approximations and is sensitive to step-size tuning.
• Penalized Q-learning achieved the highest average utilization (2.73 hours) but suffered a 28% revenue gap, indicating a trade-off between utilization and revenue.
• The proposed method outperformed both penalized Q-learning and Lagrangian actor-critic in balancing revenue and constraint compliance.
• The algorithm successfully solved medium-scale problems with up to 50 vehicles, 20 stations, and a 12-hour horizon using CPLEX for the BILP subproblems.
• The theoretical convergence of the algorithm to the optimal CMDP solution is established under mild assumptions, providing a solid foundation for practical deployment.