[Paper Review] Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?

Rishi Bommasani, Kathleen Creel+3 more|arXiv (Cornell University)|Nov 25, 2022

Ethics and Social Impacts of AI27 citations

TL;DR

The paper formalizes outcome homogenization as a risk of algorithmic monoculture and empirically tests how sharing training data and foundation models influences homogeneous negative outcomes for individuals and groups across fairness benchmarks, vision, and language tasks.

ABSTRACT

As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e.g. training data), are deployed by multiple decision-makers. While sharing offers clear advantages (e.g. amortizing costs), does it bear risks? We introduce and formalize one such risk, outcome homogenization: the extent to which particular individuals or groups experience negative outcomes from all decision-makers. If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy. To relate algorithmic monoculture and outcome homogenization, we propose the component-sharing hypothesis: if decision-makers share components like training data or specific models, then they will produce more homogeneous outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that sharing training data reliably exacerbates homogenization, with individual-level effects generally exceeding group-level effects. Further, given the dominant paradigm in AI of foundation models, i.e. models that can be adapted for myriad downstream tasks, we test whether model sharing homogenizes outcomes across tasks. We observe mixed results: we find that for both vision and language settings, the specific methods for adapting a foundation model significantly influence the degree of outcome homogenization. We conclude with philosophical analyses of and societal challenges for outcome homogenization, with an eye towards implications for deployed machine learning systems.

Motivation & Objective

• Motivate and formalize the risk of outcome homogenization as a form of systemic harm under algorithmic monoculture.
• Propose and operationalize a mathematical framework to measure homogenization at individual and group levels.
• Empirically test the component-sharing hypothesis by analyzing data sharing and foundation-model sharing across benchmarks.
• Highlight philosophical and societal implications for deployed machine learning systems.

Proposed method

• Define failures F^i for each decision-maker model h^i and the systemic failure as the probability that all models fail on an individual.
• Introduce the individual homogenization metric H^{individual} = systemic failure / product of fail(h^i) to separate homogenization from overall accuracy.
• Extend to group homogenization H_{G}^{group} with weighting schemes (average, uniform, worst).
• Experiment with data sharing using fixed vs disjoint training data across tasks and model families to compare homogenization under shared versus non-shared data.
• Experiment with foundation-model-based adaptation methods (vision: scratch, linear probing, finetuning; language: linear probing, finetuning, BitFit) to assess their impact on homogenization across tasks.

Experimental results

Research questions

• RQ1 Does sharing training data among decision-makers increase outcome homogenization for individuals and groups?
• RQ2 Do foundation-model sharing and different adaptation methods amplify or mitigate outcome homogenization across vision and language tasks?
• RQ3 How do individual-level and group-level homogenization compare, and what are the implications for fairness analyses?
• RQ4 What is the relationship between homogenization and traditional metrics like accuracy, fairness, and robustness?
• RQ5 What philosophical and societal challenges arise from outcome homogenization in deployed systems?

Key findings

• Data sharing increases outcome homogenization; fixed sharing (same data) yields more homogenization than disjoint sharing across datasets and models.
• Individual-level homogenization exceeds group-level homogenization in the ACS PUMS experiments, suggesting individuals can experience systemic harms even when group-level effects seem muted.
• Foundation-model sharing yields mixed results; the degree and mechanism of task adaptation (e.g., probing vs finetuning) significantly influence homogenization.
• Linear probing tends to produce more homogenized outcomes than finetuning in vision tasks, while language results show probing often more homogeneous than finetuning/BitFit; scratch models can be most homogeneous in some vision settings.
• A strong emphasis on individual-centric analysis is warranted, as group-level analyses may overlook systemic harms affecting specific individuals.