[Paper Review] Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning

Yongchan Kwon, James Zou|arXiv (Cornell University)|Oct 26, 2021

Machine Learning and Data Classification36 references21 citations

TL;DR

This paper proposes Beta Shapley, a unified and noise-reduced data valuation framework that generalizes Data Shapley by relaxing the efficiency axiom of the Shapley value. By weighting marginal contributions based on a beta distribution—particularly favoring small cardinalities—it achieves lower noise and superior performance in detecting mislabeled data, subsampling, and point addition/removal tasks, outperforming Data Shapley and state-of-the-art methods on multiple benchmarks.

ABSTRACT

Data Shapley has recently been proposed as a principled framework to quantify the contribution of individual datum in machine learning. It can effectively identify helpful or harmful data points for a learning algorithm. In this paper, we propose Beta Shapley, which is a substantial generalization of Data Shapley. Beta Shapley arises naturally by relaxing the efficiency axiom of the Shapley value, which is not critical for machine learning settings. Beta Shapley unifies several popular data valuation methods and includes data Shapley as a special case. Moreover, we prove that Beta Shapley has several desirable statistical properties and propose efficient algorithms to estimate it. We demonstrate that Beta Shapley outperforms state-of-the-art data valuation methods on several downstream ML tasks such as: 1) detecting mislabeled training data; 2) learning with subsamples; and 3) identifying points whose addition or removal have the largest positive or negative impact on the model.

Motivation & Objective

• To address the limitations of Data Shapley, particularly its high variance due to uniform averaging over all cardinalities.
• To develop a unified framework that generalizes multiple data valuation methods, including LOO and Data Shapley, under a single principled approach.
• To improve statistical robustness and performance in practical ML tasks such as noisy label detection and active data selection.
• To demonstrate that marginal contributions at small cardinalities carry higher signal-to-noise ratios, justifying a non-uniform weighting scheme.
• To provide an efficient Monte Carlo-based algorithm for scalable estimation of Beta Shapley values in large-scale settings.

Proposed method

• Beta Shapley generalizes Data Shapley by relaxing the efficiency axiom, allowing for a flexible weighting scheme over marginal contributions.
• It uses a beta distribution to assign weights to marginal contributions across different subset cardinalities, with parameters α and β controlling the emphasis on small or large cardinalities.
• The framework is derived as a semivalue, a generalization of the Shapley value that preserves fairness and symmetry while discarding efficiency.
• An efficient Monte Carlo sampling algorithm is proposed to estimate Beta Shapley values by sampling subsets of varying sizes with probability proportional to the beta weight.
• The method supports both positive and negative data values, enabling identification of harmful or noisy data points.
• Theoretical analysis shows that Beta Shapley has reduced statistical noise compared to Data Shapley, especially when small-cardinality contributions are prioritized.

Experimental results

Research questions

• RQ1 Can relaxing the efficiency axiom of the Shapley value lead to a more robust and effective data valuation framework in machine learning?
• RQ2 Do marginal contributions at small cardinalities provide higher signal-to-noise ratios than those at large cardinalities?
• RQ3 Can a unified framework based on beta-weighted marginal contributions outperform existing methods like Data Shapley and LOO in real-world ML tasks?
• RQ4 What is the optimal weighting scheme for marginal contributions in data valuation, and how does it depend on the ML task or data distribution?
• RQ5 How can Beta Shapley be efficiently estimated at scale using Monte Carlo sampling while maintaining statistical accuracy?

Key findings

• Beta(16,1), which emphasizes small cardinality marginal contributions, achieves the highest performance in detecting mislabeled data and point addition experiments, with an average accuracy gain of 0.757 across 15 datasets.
• Beta(16,1) outperforms Data Shapley (Beta(1,1)) and LOO in point addition tasks, showing a faster and more consistent improvement in model accuracy when selecting high-value data points.
• Data Shapley performs slightly better than other methods in point removal tasks, as the uniform weighting captures the impact of large-cardinality removals more effectively.
• The method achieves a 0.872 ± 0.004 F1 score on the German dataset, outperforming LOO and other baselines in noisy label detection.
• Beta Shapley with Beta(4,1) and Beta(16,1) shows the best performance in subsampling, achieving 0.772 ± 0.004 accuracy on the Vehicle dataset, significantly outperforming uniform sampling.
• The heatmap in Figure 7 confirms that Beta(16,1) is the top-performing method across 12 out of 15 datasets in detection and subsampling tasks, with a linearly scaled frequency of 0.757 for best performance.