[论文解读] Autonomous Airline Revenue Management: A Deep Reinforcement Learning Approach to Seat Inventory Control and Overbooking

Syed Arbab Mohd Shihab, Caleb Logemann+2 more|arXiv (Cornell University)|Feb 18, 2019

Aviation Industry Analysis and Trends参考文献 11被引用 16

一句话总结

本文提出一种基于深度Q网络（Deep Q-Networks）的深度强化学习方法，用于解决航空座位库存控制与超售问题，将其建模为马尔可夫决策过程。该方法在多种模拟市场条件下实现了接近最优的收益表现，展示了在大规模连续状态空间中对多 fare 类乘客同时到达及随机取消情况下的强大泛化能力。

ABSTRACT

Revenue management can enable airline corporations to maximize the revenue generated from each scheduled flight departing in their transportation network by means of finding the optimal policies for differential pricing, seat inventory control and overbooking. As different demand segments in the market have different Willingness-To-Pay (WTP), airlines use differential pricing, booking restrictions, and service amenities to determine different fare classes or products targeted at each of these demand segments. Because seats are limited for each flight, airlines also need to allocate seats for each of these fare classes to prevent lower fare class passengers from displacing higher fare class ones and set overbooking limits in anticipation of cancellations and no-shows such that revenue is maximized. Previous work addresses these problems using optimization techniques or classical Reinforcement Learning methods. This paper focuses on the latter problem - the seat inventory control problem - casting it as a Markov Decision Process to be able to find the optimal policy. Multiple fare classes, concurrent continuous arrival of passengers of different fare classes, overbooking and random cancellations that are independent of class have been considered in the model. We have addressed this problem using Deep Q-Learning with the goal of maximizing the reward for each flight departure. The implementation of this technique allows us to employ large continuous state space but also presents the potential opportunity to test on real time airline data. To generate data and train the agent, a basic air-travel market simulator was developed. The performance of the agent in different simulated market scenarios was compared against theoretically optimal solutions and was found to be nearly close to the expected optimal revenue.

研究动机与目标

• 通过在动态需求下对多个票价等级的座位分配进行优化，解决最大化航空收益的挑战。
• 整合超售策略，以应对取消和未到旅客的情况，同时保持高票价等级的容量。
• 为典型真实航空运营中常见的大规模连续状态空间，开发可扩展的解决方案。
• 使用深度强化学习训练一个自主智能体，实现实时动态调整库存决策。
• 在多样化模拟市场场景下，与理论最优解对比评估性能表现。

提出的方法

• 将座位库存控制与超售问题建模为具有连续状态空间的马尔可夫决策过程（MDP）。
• 实现一个深度Q网络（DQN）智能体，通过与自定义航空旅行市场模拟器的试错式交互，学习最优座位分配策略。
• 在DQN框架中使用经验回放和目标网络，以稳定训练并提升收敛性。
• 对来自不同票价等级、具有不同支付意愿（WTP）分布的乘客进行并发、连续的到达建模。
• 引入与票价等级无关的随机取消，以模拟现实中的未到旅客行为。
• 使用捕捉真实市场动态与需求模式的模拟器生成的合成数据对智能体进行训练。

实验结果

研究问题

• RQ1 深度强化学习智能体是否能够在具备多票价等级与超售能力的航空收益管理环境中，学习到接近最优的座位库存控制策略？
• RQ2 在不同需求与取消情景下，基于DQN的智能体性能与理论最优解相比如何？
• RQ3 DQN智能体在具有连续状态空间与并发乘客到达的多样化市场条件下，其泛化能力达到何种程度？
• RQ4 该智能体在平衡收益获取与超售风险及容量替代方面表现如何？
• RQ5 随机取消对智能体在动态需求下维持高收益能力有何影响？

主要发现

• DQN智能体在所有测试市场情景下均实现了接近最优的收益表现，与理论最优上界极为接近。
• 该模型成功管理了多个票价等级，并在保持高收益效率的同时，处理了并发、连续的乘客到达。
• 智能体通过动态调整超售上限，有效应对随机取消与未到旅客，显著降低了超售风险。
• 该方法在大规模连续状态空间中表现出良好的可扩展性，实现了无需显式状态离散化的实时决策。
• 在不同需求分布与预订模式下，性能保持一致，表明其具备强大的泛化能力。
• 模拟器生成的训练数据使智能体能够学习到预订行为、票价等级与收益结果之间复杂的非线性关系。