[Paper Review] Selection Problems in the Presence of Implicit Bias

Jon Kleinberg, Manish Raghavan|arXiv (Cornell University)|Jan 4, 2018

Names, Identity, and Discrimination Research21 citations

TL;DR

This paper proposes a theoretical model to analyze how implicit bias affects selection decisions in hiring, particularly evaluating the Rooney Rule—a policy requiring at least one candidate from an underrepresented group to be interviewed. It shows that under specific conditions involving bias magnitude, minority representation, and candidate quality distribution, the Rooney Rule can improve both diversity and organizational payoff by correcting for biased evaluations.

ABSTRACT

Over the past two decades, the notion of implicit bias has come to serve as an important component in our understanding of discrimination in activities such as hiring, promotion, and school admissions. Research on implicit bias posits that when people evaluate others -- for example, in a hiring context -- their unconscious biases about membership in particular groups can have an effect on their decision-making, even when they have no deliberate intention to discriminate against members of these groups. A growing body of experimental work has pointed to the effect that implicit bias can have in producing adverse outcomes. Here we propose a theoretical model for studying the effects of implicit bias on selection decisions, and a way of analyzing possible procedural remedies for implicit bias within this model. A canonical situation represented by our model is a hiring setting: a recruiting committee is trying to choose a set of finalists to interview among the applicants for a job, evaluating these applicants based on their future potential, but their estimates of potential are skewed by implicit bias against members of one group. In this model, we show that measures such as the Rooney Rule, a requirement that at least one of the finalists be chosen from the affected group, can not only improve the representation of this affected group, but also lead to higher payoffs in absolute terms for the organization performing the recruiting. However, identifying the conditions under which such measures can lead to improved payoffs involves subtle trade-offs between the extent of the bias and the underlying distribution of applicant characteristics, leading to novel theoretical questions about order statistics in the presence of probabilistic side information.

Motivation & Objective

• To formalize the impact of implicit bias on selection decisions in hiring and similar processes.
• To analyze whether procedural remedies like the Rooney Rule can improve both representation and organizational outcomes.
• To identify the conditions under which the Rooney Rule leads to higher expected quality of selected candidates despite bias.
• To explore the interplay between bias strength, minority representation, and candidate quality distributions in selection outcomes.
• To derive theoretical conditions under which the Rooney Rule enhances both fairness and performance in selection processes.

Proposed method

• Models a hiring scenario where evaluators have implicit bias against a specific group, distorting their assessment of candidates' potential.
• Uses order statistics from i.i.d. random variables to model the distribution of candidate evaluations, with one group subject to bias.
• Introduces a formal condition involving three parameters: bias magnitude, minority prevalence, and the threshold for candidate quality.
• Applies probabilistic bounds on order statistics (e.g., Pr[X_{(n-1:n)} ≥ T ∩ Y_{(n-1:n)} ≥ T] ≤ 2nF(T)^{n-1}) to analyze the likelihood of strong minority candidates being overlooked.
• Uses conditional expectation analysis to compare the expected quality of selected candidates with and without the Rooney Rule.
• Derives a lower bound on the expected improvement in candidate quality under the Rooney Rule using a truncated distribution model for high-performing candidates.

Experimental results

Research questions

• RQ1 Under what conditions does the Rooney Rule improve the expected quality of selected candidates in the presence of implicit bias?
• RQ2 How does the interplay between bias strength, minority representation, and candidate quality distribution affect the effectiveness of the Rooney Rule?
• RQ3 Can the Rooney Rule lead to higher organizational payoffs even when the majority group has stronger candidates, due to bias-induced errors?
• RQ4 What role does probabilistic side information (e.g., quality thresholds) play in determining the success of diversity-preserving selection rules?
• RQ5 Is there a sharp threshold in bias magnitude and representation where the Rooney Rule transitions from being beneficial to detrimental?

Key findings

• The Rooney Rule can increase the expected quality of selected candidates when the bias against a minority group is sufficiently strong and the pool of qualified minority candidates is large enough.
• A sharp threshold exists where the Rooney Rule improves outcomes only when the product of bias strength and minority prevalence exceeds a critical value tied to the distribution of candidate quality.
• The expected improvement in candidate quality under the Rooney Rule is bounded below by a function proportional to (F(T) + η)^{n-1}, where F(T) is the cumulative distribution of candidate quality at threshold T.
• The model shows that even when the majority group has stronger candidates on average, the Rooney Rule can correct for bias-induced errors and lead to better overall selection outcomes.
• The analysis reveals that the effectiveness of the Rooney Rule depends not only on bias and representation but also on the shape of the underlying quality distribution, particularly the tail behavior.
• Theoretical bounds on order statistics (e.g., Pr[X_{(n-1:n)} ≥ T ∩ Y_{(n-1:n)} ≥ T] ≤ 2nF(T)^{n-1}) are essential for quantifying the risk of overlooking high-quality minority candidates.