[Paper Review] Beyond Parity: Fairness Objectives for Collaborative Filtering

Sirui Yao, Bert Huang|arXiv (Cornell University)|May 24, 2017

Ethics and Social Impacts of AISocial Sciences16 references154 citations

TL;DR

The paper defines four new fairness metrics for collaborative filtering, integrates them as regularizers in matrix factorization, and shows in synthetic and real data that these metrics can be minimized with little loss in accuracy, with trade-offs among unfairness types.

ABSTRACT

We study fairness in collaborative-filtering recommender systems, which are sensitive to discrimination that exists in historical data. Biased data can lead collaborative-filtering methods to make unfair predictions for users from minority groups. We identify the insufficiency of existing fairness metrics and propose four new metrics that address different forms of unfairness. These fairness metrics can be optimized by adding fairness terms to the learning objective. Experiments on synthetic and real data show that our new metrics can better measure fairness than the baseline, and that the fairness objectives effectively help reduce unfairness.

Motivation & Objective

• Motivate and formalize unfairness in collaborative-filtering recommender systems beyond demographic parity.
• Introduce four distinct fairness metrics capturing different forms of unfairness in predictions.
• Propose and optimize learning objectives that incorporate fairness penalties into matrix-factorization.
• Evaluate how these fairness objectives affect prediction accuracy and various unfairness measures on synthetic and real data.

Proposed method

• Use matrix factorization for collaborative filtering with bias terms as in r_{ij} ≈ p_i^T q_j + u_i + v_j.
• Define two forms of data imbalance: population imbalance and observation bias via stochastic block models.
• Introduce four unfairness metrics: value unfairness, absolute unfairness, underestimation unfairness, and overestimation unfairness; plus a non-parity baseline U_par.
• Augment the learning objective with a smoothed fairness penalty U (Huber-like) to optimize a trade-off between accuracy and fairness.
• Employ gradient-based optimization (Adam) to train with fairness penalties and evaluate on synthetic and Movielens data.

Experimental results

Research questions

• RQ1 How does data underrepresentation (population imbalance and observation bias) affect fairness in collaborative filtering?
• RQ2 Can new fairness metrics better capture different unfairness forms than demographic parity in recommender systems?
• RQ3 Do fairness-augmented learning objectives reduce unfairness across metrics with limited degradation in predictive accuracy?
• RQ4 How do different fairness objectives interact and trade off with each other in practice?

Key findings

• Unfairness arises from underrepresentation even when observed ratings reflect true preferences.
• The four proposed metrics measure distinct unfairness aspects and respond differently to data bias.
• Optimizing any fairness metric generally reduces other unfairness types as well, with some trade-offs.
• Fairness-augmented matrix factorization can reduce unfairness with no significant increase in reconstruction error across synthetic and real data; some metrics (e.g., value fairness) also reduce under- and overestimation.
• Non-parity tends to increase or not improve other unfairness metrics, highlighting limitations of demographic parity-like constraints in recommendations.
• A combined objective that penalizes both overestimation and underestimation provides a practical approach, though no single objective dominates across all metrics.