[Paper Review] Absolute Shapley Value

Jinfei Liu|arXiv (Cornell University)|Mar 23, 2020

Bayesian Modeling and Causal Inference8 references4 citations

TL;DR

This paper introduces Absolute Shapley Value, a novel method for evaluating data importance in machine learning by taking the absolute value of marginal contributions, even when they are negative. Experiments on the Iris dataset show that Absolute Shapley Value outperforms Original and Zero Shapley Value in identifying truly important or unimportant data tuples, as evidenced by superior model accuracy when training on top-k data based on Shapley values.

ABSTRACT

Shapley value is a concept in cooperative game theory for measuring the contribution of each participant, which was named in honor of Lloyd Shapley. Shapley value has been recently applied in data marketplaces for compensation allocation based on their contribution to the models. Shapley value is the only value division scheme used for compensation allocation that meets three desirable criteria: group rationality, fairness, and additivity. In cooperative game theory, the marginal contribution of each contributor to each coalition is a nonnegative value. However, in machine learning model training, the marginal contribution of each contributor (data tuple) to each coalition (a set of data tuples) can be a negative value, i.e., the accuracy of the model trained by a dataset with an additional data tuple can be lower than the accuracy of the model trained by the dataset only. In this paper, we investigate the problem of how to handle the negative marginal contribution when computing Shapley value. We explore three philosophies: 1) taking the original value (Original Shapley Value); 2) taking the larger of the original value and zero (Zero Shapley Value); and 3) taking the absolute value of the original value (Absolute Shapley Value). Experiments on Iris dataset demonstrate that the definition of Absolute Shapley Value significantly outperforms the other two definitions in terms of evaluating data importance (the contribution of each data tuple to the trained model).

Motivation & Objective

• To address the challenge of negative marginal contributions in data valuation for machine learning models.
• To evaluate whether taking absolute values of marginal contributions improves data importance assessment.
• To compare three philosophies of Shapley value definition: Original, Zero, and Absolute.
• To empirically validate which definition best identifies important data tuples in terms of model performance.
• To provide a theoretically sound and practically effective method for data valuation in data marketplaces and model training.

Proposed method

• Proposes Absolute Shapley Value as a new definition where the marginal contribution of each data tuple is replaced by its absolute value in the Shapley value computation.
• Uses Monte Carlo sampling to approximate the Shapley value efficiently, averaging marginal contributions over random permutations of data tuples.
• Defines three variants: Original (raw marginal contribution), Zero (clipped at zero), and Absolute (absolute value of marginal contribution).
• Employs a standard utility function based on model accuracy (SVM and Logistic Regression) evaluated on a held-out test set.
• Applies the Shapley value computation to the Iris dataset, using top-K and bottom-K data tuples to train models and measure accuracy.
• Employs algorithmic sampling to estimate Shapley values with convergence control, ensuring unbiased and efficient approximation.

Experimental results

Research questions

• RQ1 Does taking the absolute value of marginal contributions improve the identification of important data tuples in machine learning models?
• RQ2 How do different definitions of Shapley value—Original, Zero, and Absolute—affect the ranking of data importance?
• RQ3 Can Absolute Shapley Value better distinguish between truly important and unimportant data tuples compared to existing definitions?
• RQ4 Does the model trained on top-K data tuples ranked by Absolute Shapley Value achieve higher accuracy than those trained on top-K by Original or Zero Shapley Value?
• RQ5 Is the Absolute Shapley Value definition more robust in capturing the true contribution of data tuples, especially when marginal contributions are negative?

Key findings

• For both Logistic Regression and SVM on the Iris dataset, the model trained on the top 35 data tuples ranked by Absolute Shapley Value achieved 100.00% accuracy, confirming its effectiveness in identifying high-value data.
• The model trained on the lowest 35 data tuples ranked by Absolute Shapley Value achieved 60.00% accuracy, significantly lower than the 63.33% achieved by the Zero Shapley definition, indicating that Absolute Shapley better identifies unimportant data.
• In contrast, the Original Shapley Value assigned equal accuracy (100.00%) to models trained on top-K and bottom-K data tuples, indicating it fails to distinguish between important and unimportant data.
• The Absolute Shapley Value definition correctly identified 6 out of 10 support vectors in the highest 10 ranked tuples, outperforming Zero (5) and Original (4), confirming better alignment with model structure.
• The lowest 10 tuples under Absolute Shapley Value were more compact and centrally located in the feature space, indicating they are less influential, which aligns with their low valuation.
• The results demonstrate that Absolute Shapley Value is superior in evaluating data importance because it treats large-magnitude negative contributions as significant, unlike Original and Zero definitions.