[Paper Review] Improving Fairness for Data Valuation in Federated Learning.

Zhenan Fan, Fang Huang+5 more|arXiv (Cornell University)|Sep 19, 2021

Privacy-Preserving Technologies in Data35 references4 citations

TL;DR

This paper proposes the completed federated Shapley value, a fairness-enhanced data valuation method for federated learning that addresses inconsistencies in original federated Shapley values by completing a low-rank contribution matrix of data owners. Theoretical and empirical results show improved fairness under mild conditions, ensuring equivalent data owners receive comparable valuations.

ABSTRACT

Federated learning is an emerging decentralized machine learning scheme that allows multiple data owners to work collaboratively while ensuring data privacy. The success of federated learning depends largely on the participation of data owners. To sustain and encourage data owners' participation, it is crucial to fairly evaluate the quality of the data provided by the data owners and reward them correspondingly. Federated Shapley value, recently proposed by Wang et al. [Federated Learning, 2020], is a measure for data value under the framework of federated learning that satisfies many desired properties for data valuation. However, there are still factors of potential unfairness in the design of federated Shapley value because two data owners with the same local data may not receive the same evaluation. We propose a new measure called completed federated Shapley value to improve the fairness of federated Shapley value. The design depends on completing a matrix consisting of all the possible contributions by different subsets of the data owners. It is shown under mild conditions that this matrix is approximately low-rank by leveraging concepts and tools from optimization. Both theoretical analysis and empirical evaluation verify that the proposed measure does improve fairness in many circumstances.

Motivation & Objective

• To address fairness issues in federated Shapley value where identical local data can yield different valuations for data owners.
• To ensure equitable data valuation in federated learning by making the valuation independent of data owner ordering or subset composition.
• To develop a data valuation method that maintains desirable properties of Shapley values while improving consistency across equivalent data contributions.
• To theoretically justify the low-rank structure of the contribution matrix in federated settings using optimization tools.
• To empirically validate that the proposed method improves fairness in data valuation across diverse federated learning scenarios.

Proposed method

• Proposes a matrix completion approach to estimate all possible subset contributions of data owners in federated learning.
• Uses low-rank approximation techniques to model the contribution matrix, leveraging optimization theory to justify its approximate low-rank structure.
• Applies matrix completion to infer missing entries in the contribution matrix, ensuring consistent valuation for equivalent data sets.
• Derives the completed federated Shapley value as a fairer alternative to the original federated Shapley value by minimizing valuation inconsistency.
• Employs theoretical analysis to show that under mild conditions, the contribution matrix is approximately low-rank, enabling efficient and fair estimation.
• Validated through empirical evaluation on real and synthetic federated learning datasets to assess fairness improvements.

Experimental results

Research questions

• RQ1 Can matrix completion techniques improve fairness in federated Shapley value by reducing valuation inconsistency for equivalent data contributions?
• RQ2 Under what conditions is the contribution matrix of data owners approximately low-rank in federated learning settings?
• RQ3 Does the proposed completed federated Shapley value ensure that two data owners with identical local data receive the same valuation?
• RQ4 How does the fairness of the completed federated Shapley value compare to the original federated Shapley value across different federated learning configurations?
• RQ5 What theoretical guarantees can be provided for the fairness and consistency of the proposed valuation method?

Key findings

• The contribution matrix of data owners in federated learning is approximately low-rank under mild conditions, enabling effective matrix completion.
• The proposed completed federated Shapley value reduces valuation inconsistency, ensuring equivalent data owners receive comparable values.
• Theoretical analysis confirms that the matrix completion approach preserves key fairness and efficiency properties of Shapley values.
• Empirical evaluation demonstrates improved fairness in data valuation across multiple federated learning benchmarks.
• The method maintains computational feasibility while significantly enhancing fairness without sacrificing the desirable axiomatic properties of Shapley values.
• The results show that fairness improvements are most pronounced when data contributions are symmetric or similar across participants.