[Paper Review] Pricing from Observational Data

Dimitris Bertsimas, Nathan Kallus|arXiv (Cornell University)|May 8, 2016

Monetary Policy and Economic Impact9 references4 citations

TL;DR

This paper proposes a causal inference-based framework for optimal pricing from observational data, addressing the limitations of traditional predictive approaches that fail to identify the true profit-maximizing price due to missing counterfactual demand. It establishes identifiability conditions, develops parametric and non-parametric algorithms with theoretical guarantees, and demonstrates in an auto loan dataset that their method recovers 36–70% of the profit lost by predictive approaches.

ABSTRACT

Given observational data on price and demand, the price optimization problem is sometimes addressed in the literature by a predictive approach: (a) fit a model to the data that best predicts demand given price and (b) substitute the predictive model into the overall profit and optimize for price. We show that, because historical demand at all prices but the observed one is missing, the price optimization problem is not well specified by the data, and in particular, the predictive approach fails to find the optimal price. We bound the suboptimality of the predictive approach, even when the optimal price cannot be identified from the data, by leveraging the special structure of the problem. Drawing from the causal inference literature, we provide su cient conditions for the optimal price to be identifiable from the data. Given these conditions, we provide parametric and non-parametric algorithms for the price optimization problem. In the non-parametric case we prove consistency and asymptotic normality and establish rates of convergence. We develop a hypothesis test for asymptotic profit optimality of any algorithm for pricing from observational data. We use this test to demonstrate empirically in an auto loan dataset that both parametric and non-parametric predictive approaches lose significant profit relative to the optimum and that our prescriptive parametric framework leads to profit that cannot be distinguished from the optimal one, recovering 36-70% of profits lost by the predictive approaches.

Motivation & Objective

• To address the fundamental flaw in predictive approaches to price optimization, which cannot identify the true optimal price due to missing counterfactual demand data.
• To establish sufficient conditions under which the optimal price is identifiable from observational data using causal inference principles.
• To develop both parametric and non-parametric algorithms for price optimization that are consistent and asymptotically normal under these conditions.
• To provide a hypothesis test for assessing the asymptotic profit optimality of any pricing algorithm from observational data.
• To empirically demonstrate the superiority of the proposed framework over standard predictive methods in real-world data.

Proposed method

• Leverages structural causal models to identify conditions under which the optimal price is identifiable from observational data.
• Proposes a non-parametric algorithm that estimates the demand function using kernel methods, with proven consistency and asymptotic normality.
• Develops a parametric framework that models demand as a function of price under structural assumptions, enabling efficient optimization.
• Introduces a hypothesis test to evaluate whether any pricing algorithm achieves asymptotic profit optimality.
• Uses the non-parametric estimator to bound suboptimality of predictive approaches even when the optimal price is not fully identifiable.
• Employs a two-stage estimation procedure: first estimate the demand function, then optimize profit using the estimated model.

Experimental results

Research questions

• RQ1 Under what conditions can the profit-maximizing price be identified from observational data on price and demand?
• RQ2 How does the suboptimality of predictive approaches compare to the true optimal price when counterfactual demand is unobserved?
• RQ3 Can a non-parametric method for demand estimation achieve consistency and asymptotic normality in the context of price optimization?
• RQ4 Is there a statistical test to determine whether a given pricing algorithm is asymptotically profit-optimal?
• RQ5 How much profit is lost by standard predictive approaches compared to the optimal solution in real-world settings?

Key findings

• The predictive approach to price optimization is fundamentally suboptimal due to unobserved counterfactual demand, even when the model fits the observed data well.
• The proposed non-parametric algorithm is consistent and asymptotically normal, with established rates of convergence for the estimated demand function.
• In an auto loan dataset, predictive approaches lose significant profit—up to 70% of the maximum achievable profit.
• The proposed parametric framework produces pricing decisions that are statistically indistinguishable from the optimal profit, recovering 36–70% of the profit lost by predictive methods.
• The proposed hypothesis test successfully identifies algorithms that are asymptotically profit-optimal, enabling empirical validation of pricing strategies.
• The study establishes that identifiability of the optimal price is possible under specific structural conditions derived from causal inference.