[Paper Review] DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation

Felipe Garrido-Lucero, Benjamin Heymann+3 more|arXiv (Cornell University)|Jun 3, 2023

Explainable Artificial Intelligence (XAI)42 references4 citations

TL;DR

This paper introduces DU-Shapley, a novel, efficient proxy for the Shapley value in dataset valuation that exploits the structure of utility functions dependent on dataset size. By approximating the Shapley value as an expectation over a discrete uniform distribution over coalition sizes, DU-Shapley reduces computational complexity and achieves almost sure convergence to the true Shapley value as the number of data owners grows, outperforming standard Monte Carlo methods even with few data owners.

ABSTRACT

We consider the dataset valuation problem, that is, the problem of quantifying the incremental gain, to some relevant pre-defined utility of a machine learning task, of aggregating an individual dataset to others. The Shapley value is a natural tool to perform dataset valuation due to its formal axiomatic justification, which can be combined with Monte Carlo integration to overcome the computational tractability challenges. Such generic approximation methods, however, remain expensive in some cases. In this paper, we exploit the knowledge about the structure of the dataset valuation problem to devise more efficient Shapley value estimators. We propose a novel approximation, referred to as discrete uniform Shapley, which is expressed as an expectation under a discrete uniform distribution with support of reasonable size. We justify the relevancy of the proposed framework via asymptotic and non-asymptotic theoretical guarantees and illustrate its benefits via an extensive set of numerical experiments.

Motivation & Objective

• To address the high computational cost of exact and Monte Carlo-based Shapley value estimation in dataset valuation.
• To develop a more efficient alternative to generic Monte Carlo approximation by leveraging the structural dependence of utility functions on dataset size.
• To provide theoretical guarantees—both asymptotic and non-asymptotic—on the approximation error of the proposed method.
• To demonstrate empirically that DU-Shapley outperforms existing Shapley value approximations, including Owen-Shapley, even with small numbers of data owners.

Proposed method

• DU-Shapley approximates the Shapley value as an expectation over a discrete uniform distribution over coalition sizes, with support size equal to the number of data owners.
• The method replaces the full enumeration of all subsets with a structured approximation that averages marginal contributions across fixed coalition sizes.
• It leverages the fact that utility functions in dataset valuation often depend on the total number of data points, enabling a simplification based on expected dataset sizes.
• The estimator is defined as $ \psi_i = \frac{1}{I} \left[ w(n_i) + w(n_\mathcal{I}) - w(n_{\mathcal{I}\setminus\{i\}}) \right] + \frac{1}{I} \sum_{k=1}^{I-2} \frac{1}{\binom{I-1}{k}} \sum_{\substack{\mathcal{S} \subseteq \mathcal{I}\setminus\{i\} \\ |\mathcal{S}|=k}} \left[ w(n_\mathcal{S} + n_i) - w(n_\mathcal{S}) \right] $, with a simplified version using expected values.
• The method includes a variant, DU-Shapley++, which approximates the sum over intermediate-sized coalitions using expected utility under average dataset sizes.
• Theoretical analysis shows that DU-Shapley converges almost surely to the true Shapley value as the number of data owners increases.

Figure 2: Monte Carlo’s expected error for limited sampling budget ( $T=I$ ) versus DU-Shapley ’s expected bias for a value function $w(n)=1-\frac{10^{k(\mathcal{I})}}{10^{k(\mathcal{I})}+n}$ where $k(\mathcal{I})=\lfloor\log(n_{\mathcal{I}})\rfloor-1$ . For each value of $I$ , we drew $100$ times t

Experimental results

Research questions

• RQ1 Can we design a more efficient Shapley value approximation for dataset valuation by exploiting the structure of the utility function?
• RQ2 Does the proposed DU-Shapley estimator converge to the true Shapley value as the number of data owners grows?
• RQ3 How does the approximation error of DU-Shapley scale with respect to key problem parameters such as average dataset size and utility function growth?
• RQ4 Can DU-Shapley outperform standard Monte Carlo and state-of-the-art approximations like Owen-Shapley in practice, even with a small number of data owners?

Key findings

• DU-Shapley achieves a significant reduction in the number of required utility evaluations compared to exact Shapley value computation, which is exponential in the number of data owners.
• The method is theoretically justified with non-asymptotic error bounds that depend explicitly on the average dataset size, its variance, and the growth rate of the utility function.
• DU-Shapley converges almost surely to the true Shapley value as the number of data owners increases, a property that generic Monte Carlo methods fail to achieve under limited sampling budgets.
• Empirical evaluations show that DU-Shapley outperforms other Shapley value approximations, including Owen-Shapley, on multiple dataset valuation benchmarks, even when the number of data owners is small.
• The variant DU-Shapley++ maintains the same computational cost as DU-Shapley but improves approximation accuracy by replacing sum-of-terms with expected values over coalition sizes.