[Paper Review] Choice-Model-Assisted Q-learning for Delayed-Feedback Revenue Management

Owen Shen, Patrick Jaillet|arXiv (Cornell University)|Feb 2, 2026

Supply Chain and Inventory Management0 citations

TL;DR

The paper proposes Choice-Model-Assisted RL (CA-DQN) that uses a fixed discrete choice model to impute delayed revenue, enabling immediate Q-learning updates; it proves convergence bounds and validates robustness and limitations via simulation on hotel bookings.

ABSTRACT

We study reinforcement learning for revenue management with delayed feedback, where a substantial fraction of value is determined by customer cancellations and modifications observed days after booking. We propose \emph{choice-model-assisted RL}: a calibrated discrete choice model is used as a fixed partial world model to impute the delayed component of the learning target at decision time. In the fixed-model deployment regime, we prove that tabular Q-learning with model-imputed targets converges to an $O(\varepsilon/(1-γ))$ neighborhood of the optimal Q-function, where $\varepsilon$ summarizes partial-model error, with an additional $O(t^{-1/2})$ sampling term. Experiments in a simulator calibrated from 61{,}619 hotel bookings (1{,}088 independent runs) show: (i) no statistically detectable difference from a maturity-buffer DQN baseline in stationary settings; (ii) positive effects under in-family parameter shifts, with significant gains in 5 of 10 shift scenarios after Holm--Bonferroni correction (up to 12.4\%); and (iii) consistent degradation under structural misspecification, where the choice model assumptions are violated (1.4--2.6\% lower revenue). These results characterize when partial behavioral models improve robustness under shift and when they introduce harmful bias.

Motivation & Objective

• Address revenue management with delayed feedback where cancellations/modifications reveal rewards days later.
• Embed a fixed discrete choice model as a partial world model to impute delayed rewards at decision time.
• Establish theoretical convergence guarantees for tabular Q-learning with model-imputed targets.
• Empirically evaluate robustness under distributional shifts and structural misspecification using a simulator built from real hotel bookings.
• Characterize when partial behavioral models improve robustness and when they introduce bias.

Proposed method

• Define delayed-feedback MDP with shocks and distinguish immediate versus delayed rewards.
• Embed a fixed pre-trained discrete choice model (DCM) as a partial world model to impute delayed rewards at decision time.
• Introduce model-imputed sampling where synthetic (r', s') samples are generated from the DCM for Q-learning updates.
• Prove finite-time convergence bounds: ||Q_t - Q*||_∞ = O(ε/(1-γ) + t^{-1/2}√log(...)) with ε capturing DCM error.
• Show an adaptive two-timescale framework where the DCM guides learning while maintaining interpretability and tractability.

Figure 3 : Learning curves in stationary settings. Both MB-DQN (orange) and Choice-Assisted DQN (blue) converge to similar performance levels across all training durations (n=20 seeds per method, shaded regions show 95% confidence intervals). No significant differences are detected ( $p>0.05$ at all

Experimental results

Research questions

• RQ1 Does CA-DQN match MB-DQN when the DCM is correctly specified in stationary settings?
• RQ2 Does CA-DQN improve robustness under in-family shifts (demand/competition) without sacrificing performance?
• RQ3 How does CA-DQN perform under structural misspecification of the DCM (IIA violations, heterogeneity, temporal dynamics)?
• RQ4 What are the theoretical convergence properties of Q-learning when using a fixed, model-imputed target?

Key findings

• CA-DQN attains convergence to a near-optimal Q-function with an irreducible bias determined by the DCM approximation error, plus a diminishing sampling term.
• In stationary settings, CA-DQN shows no statistically significant difference from MB-DQN, indicating practical equivalence when the DCM is correct.
• CA-DQN improves robustness under several in-family shifts, with up to 12.4% gains in some scenarios after correction for multiple comparisons.
• Under structural misspecification, CA-DQN degrades consistently (1.4%–2.6% lower revenue in misspecification tests).
• Experiments on a simulator calibrated from 61,619 hotel bookings show the outlined trade-offs between robustness and bias.

Figure 4 : Robustness under parameter shifts across 10 scenarios. Choice-Assisted DQN (blue bars) shows mixed results compared to MB-DQN (orange bars): significant improvements in 4 scenarios (up to +12.4% under low demand), significant underperformance in 2 scenarios (up to -9.6% under high competi