[Paper Review] Decision Trees for Decision-Making under the Predict-then-Optimize Framework

Adam N. Elmachtoub, Jason Cheuk Nam Liang+1 more|arXiv (Cornell University)|Feb 29, 2020

Explainable Artificial Intelligence (XAI)Computer Science47 citations

TL;DR

The paper introduces SPO Trees (SPOTs) to train decision trees directly on the SPO loss for predict-then-optimize problems, achieving higher-quality decisions with lower model complexity than traditional prediction-error-focused trees.

ABSTRACT

We consider the use of decision trees for decision-making problems under the predict-then-optimize framework. That is, we would like to first use a decision tree to predict unknown input parameters of an optimization problem, and then make decisions by solving the optimization problem using the predicted parameters. A natural loss function in this framework is to measure the suboptimality of the decisions induced by the predicted input parameters, as opposed to measuring loss using input parameter prediction error. This natural loss function is known in the literature as the Smart Predict-then-Optimize (SPO) loss, and we propose a tractable methodology called SPO Trees (SPOTs) for training decision trees under this loss. SPOTs benefit from the interpretability of decision trees, providing an interpretable segmentation of contextual features into groups with distinct optimal solutions to the optimization problem of interest. We conduct several numerical experiments on synthetic and real data including the prediction of travel times for shortest path problems and predicting click probabilities for news article recommendation. We demonstrate on these datasets that SPOTs simultaneously provide higher quality decisions and significantly lower model complexity than other machine learning approaches (e.g., CART) trained to minimize prediction error.

Motivation & Objective

• Motivate decision-making problems where uncertain inputs are predicted before optimization.
• Propose a tractable method to train decision trees under the SPO loss rather than prediction error.
• Provide algorithms (greedy/recursive and MILP-based) to construct SPOTs.
• Show that SPOTs yield higher-quality decisions with simpler trees compared to CART and similar approaches.

Proposed method

• Model the decision problem as minimizing c^T w over a feasible set S with c predicted from features x.
• Use SPO loss to measure decision suboptimality rather than prediction error.
• Prove that the leaf-wise SPO loss is minimized by predicting the leaf-mean cost c̄_l (Theorem 4.1).
• Develop SPOTs via (a) recursive partitioning (greedy) and (b) mixed-integer linear programming (MILP) formulations.
• Train ensembles (SPOT Forests) to boost decision quality.
• Provide an open-source implementation at the authors’ repository.
• Leverage an illustrative shortest-path example to contrast SPO loss with MSE loss (CART).

Experimental results

Research questions

• RQ1 Can decision trees be trained directly to minimize SPO loss rather than prediction error?
• RQ2 How can SPO loss be incorporated into tractable tree-building procedures (greedy and MILP)?
• RQ3 Do SPOTs yield better decision quality and lower model complexity than CART under the same data?
• RQ4 Can SPOTs be extended to ensembles to further improve prescriptive performance?

Key findings

• SPOTs produce higher quality decisions and substantially lower leaf/node complexity than CART when optimizing under SPO loss.
• The average leaf cost vector minimizes within-leaf SPO loss (closed-form leaf predictor c̄_l under uniqueness of the optimal decision).
• Greedy recursive partitioning efficiently solves the SPO-tree training objective by evaluating splits via the SPO loss in leaves (using Theorem 4.1).
• An MILP formulation (building on existing tree-encodings) can solve SPO-tree training to optimality for modest problem sizes.
• SPOT Forests (ensembles) further enhance decision performance without a proportional increase in interpretability.