[Paper Review] Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration

Julian Rodemann, Federico Croppi+8 more|arXiv (Cornell University)|Mar 7, 2024

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces ShapleyBO, a framework that uses Shapley values to explain Bayesian optimization (BO) proposals by quantifying each hyperparameter's contribution to the acquisition function. By decomposing exploration into aleatoric and epistemic uncertainty components and enabling human-in-the-loop intervention, ShapleyBO improves human-AI collaboration, reducing regret in personalized exosuit tuning by up to 30% compared to non-explainable BO.

ABSTRACT

Bayesian optimization (BO) with Gaussian processes (GP) has become an indispensable algorithm for black box optimization problems. Not without a dash of irony, BO is often considered a black box itself, lacking ways to provide reasons as to why certain parameters are proposed to be evaluated. This is particularly relevant in human-in-the-loop applications of BO, such as in robotics. We address this issue by proposing ShapleyBO, a framework for interpreting BO's proposals by game-theoretic Shapley values.They quantify each parameter's contribution to BO's acquisition function. Exploiting the linearity of Shapley values, we are further able to identify how strongly each parameter drives BO's exploration and exploitation for additive acquisition functions like the confidence bound. We also show that ShapleyBO can disentangle the contributions to exploration into those that explore aleatoric and epistemic uncertainty. Moreover, our method gives rise to a ShapleyBO-assisted human machine interface (HMI), allowing users to interfere with BO in case proposals do not align with human reasoning. We demonstrate this HMI's benefits for the use case of personalizing wearable robotic devices (assistive back exosuits) by human-in-the-loop BO. Results suggest human-BO teams with access to ShapleyBO can achieve lower regret than teams without.

Motivation & Objective

• To address the black-box nature of Bayesian optimization (BO) in human-in-the-loop applications, especially in robotics.
• To enable human users to understand and trust BO's parameter proposals by providing feature-level explanations via Shapley values.
• To disentangle the contributions of hyperparameters to exploration (uncertainty reduction) and exploitation (mean optimization) in additive acquisition functions.
• To develop a ShapleyBO-assisted human-machine interface (HMI) that allows users to intervene when proposals conflict with human reasoning.
• To empirically validate that ShapleyBO enhances human-AI collaboration in real-world personalization tasks, such as tuning wearable exosuits.

Proposed method

• Shapley values are applied to the acquisition function to quantify each hyperparameter's contribution to BO's decision-making.
• The linearity of Shapley values enables decomposition of the acquisition function into contributions from exploration and exploitation components.
• Aleatoric and epistemic uncertainty contributions are further disentangled using a set-valued prior to estimate model uncertainty.
• A human-machine interface (HMI) is designed to visualize Shapley values and allow users to reject or modify BO proposals based on interpretability.
• The method is evaluated in a simulated human-in-the-loop BO setup for personalizing assistive back exosuits across 15 individuals.
• ShapleyBO is implemented with code and results reproducible via an anonymous GitHub repository.

Figure 1: ShapleyBO results in iteration 59 of BO on $\Psi(\bm{\theta})=f(\bm{\theta})+\bm{\epsilon}$ . Plots i and ii for $\lambda=1$ and plots iii and iv for $\lambda=10$ , see (Croppi,, 2021 , Fig. 5) . Contributions ( $phi$ ) are averaged over 30 proposals for each $\lambda$ . On the right, the

Experimental results

Research questions

• RQ1 How can Shapley values be used to explain the contributions of individual hyperparameters to Bayesian optimization’s acquisition function?
• RQ2 To what extent can Shapley values disentangle the roles of exploration and exploitation in additive acquisition functions like the confidence bound?
• RQ3 Can Shapley values further decompose exploration into contributions from aleatoric and epistemic uncertainty?
• RQ4 Does providing Shapley-based explanations improve human-AI collaboration in hyperparameter tuning tasks?
• RQ5 What is the impact of ShapleyBO-assisted HMI on optimization performance compared to standard BO?

Key findings

• ShapleyBO successfully explains BO’s parameter proposals by quantifying each hyperparameter’s contribution to the acquisition function using game-theoretic Shapley values.
• The method enables dissection of exploration into contributions from aleatoric and epistemic uncertainty, with epistemic uncertainty further partitioned into approximation and model uncertainty.
• In human-in-the-loop exosuit personalization, ShapleyBO-assisted teams achieved significantly lower regret than teams without interpretability, with 10 out of 15 individuals showing statistical significance at the 95% confidence level.
• The ShapleyBO-assisted HMI allowed users to reject proposals inconsistent with human reasoning, improving alignment between human intuition and algorithmic decisions.
• The framework demonstrated that Shapley values are effective for interpreting not only model predictions but also the decision logic of optimization algorithms.
• Code and results are publicly available, enabling reproducibility and extension of the framework to other BO applications.

Figure 2: Informativeness paths for hyper ellipsoid optimization, see (Croppi,, 2021 , Fig. 6) . Plot on top displays $cb$ contributions with facets for parameters (vertical) and $\lambda$ (horizontal); beneath its decomposition into $m$ (red) and $se$ (blue) contributions. They are averaged over 30