[Paper Review] Towards Efficient Data Valuation Based on the Shapley Value

Ruoxi Jia, David Dao+8 more|arXiv (Cornell University)|Feb 27, 2019

Game Theory and Voting SystemsEconomics, Econometrics and Finance185 citations

TL;DR

The paper studies data valuation using the Shapley value and develops efficient approximation algorithms to estimate data contributions for large ML datasets, with applicability to fair compensation and data-market designs.

ABSTRACT

"How much is my data worth?" is an increasingly common question posed by organizations and individuals alike. An answer to this question could allow, for instance, fairly distributing profits among multiple data contributors and determining prospective compensation when data breaches happen. In this paper, we study the problem of data valuation by utilizing the Shapley value, a popular notion of value which originated in cooperative game theory. The Shapley value defines a unique payoff scheme that satisfies many desiderata for the notion of data value. However, the Shapley value often requires exponential time to compute. To meet this challenge, we propose a repertoire of efficient algorithms for approximating the Shapley value. We also demonstrate the value of each training instance for various benchmark datasets.

Motivation & Objective

• Motivate fair valuation of data contributions in ML setups using the Shapley value.
• Address the computational challenge of SV calculation on large datasets.
• Develop scalable approximation algorithms with provable guarantees under various assumptions.
• Explore practical estimators for ML tasks leveraging stability, smoothness, and incremental training.
• Provide empirical validation on benchmark datasets to demonstrate feasibility and scalability.

Proposed method

• Formulate data valuation as a cooperative game where data points are players and the utility is dataset-driven model performance.
• Develop a baseline permutation sampling method with probabilistic guarantees for SV estimation.
• Introduce a group testing-based SV estimation algorithm to reduce utility evaluations.
• Apply compressive sensing ideas (random Bernoulli measurements) to exploit potential sparsity in data values for SV recovery.
• Leverage stability of learning algorithms to justify uniform value division as an approximation.
• Incorporate influence-function-based heuristics to accelerate SV estimation for ML models.

Experimental results

Research questions

• RQ1 Can the Shapley value be efficiently approximated for data valuation in large ML datasets with provable guarantees?
• RQ2 How do different assumptions (bounded utility, monotone/sparse value, stability, smooth loss) affect the efficiency and accuracy of SV estimation?
• RQ3 What practical estimators (group testing, compressive sensing, influence functions) perform well across common ML tasks?
• RQ4 How does SV-based data valuation relate to privacy and adversarial robustness in practice?

Key findings

• An O(N(log N)^2) model evaluations algorithm can approximate SV with provable error guarantees under bounded utility.
• If the utility is monotone and SV is sparse, the number of model trainings can be reduced to O(log log N) with incremental maintenance.
• Group testing-based SV estimation significantly reduces utility evaluations compared to baseline permutation sampling.
• Compressive permutation sampling can recover the SV with O(N log N log log N) evaluations under sparsity assumptions.
• Influence-function-based heuristics can accelerate SV estimation, especially when leveraging largest-subset approximations.
• Stable learning algorithms yield near-uniform data values, supporting uniform value division as a practical approximation.