[Paper Review] Causal effect of racial bias in data and machine learning algorithms on user persuasiveness & discriminatory decision making: An Empirical Study

Kinshuk Sengupta, Praveen Ranjan Srivastava|arXiv (Cornell University)|Jan 22, 2022

Ethics and Social Impacts of AI4 citations

TL;DR

This empirical study investigates how racial bias in training data and machine learning models affects user persuasiveness and decision-making. Using controlled lab experiments with counterfactual analysis, the study demonstrates that biased AI models reduce user persuasiveness and distort decision-making, highlighting the need for ethical AI design to ensure fairness and trustworthiness in NLP systems.

ABSTRACT

Language data and models demonstrate various types of bias, be it ethnic, religious, gender, or socioeconomic. AI/NLP models, when trained on the racially biased dataset, AI/NLP models instigate poor model explainability, influence user experience during decision making and thus further magnifies societal biases, raising profound ethical implications for society. The motivation of the study is to investigate how AI systems imbibe bias from data and produce unexplainable discriminatory outcomes and influence an individual's articulateness of system outcome due to the presence of racial bias features in datasets. The design of the experiment involves studying the counterfactual impact of racial bias features present in language datasets and its associated effect on the model outcome. A mixed research methodology is adopted to investigate the cross implication of biased model outcome on user experience, effect on decision-making through controlled lab experimentation. The findings provide foundation support for correlating the implication of carry-over an artificial intelligence model solving NLP task due to biased concept presented in the dataset. Further, the research outcomes justify the negative influence on users' persuasiveness that leads to alter the decision-making quotient of an individual when trying to rely on the model outcome to act. The paper bridges the gap across the harm caused in establishing poor customer trustworthiness due to an inequitable system design and provides strong support for researchers, policymakers, and data scientists to build responsible AI frameworks within organizations.

Motivation & Objective

• To examine the causal impact of racial bias in language datasets on user persuasiveness and decision-making outcomes.
• To investigate how biased AI models influence individual decision-making when users rely on model-generated outputs.
• To assess the role of model explainability and bias in undermining user trust and fairness in AI-assisted decisions.
• To provide empirical evidence linking biased data to discriminatory outcomes and reduced user articulateness in AI interactions.
• To support the development of responsible AI frameworks by identifying the harms of biased model design in real-world applications.

Proposed method

• Employed a mixed-methods research design combining quantitative lab experiments with qualitative analysis of user behavior.
• Conducted controlled experiments using counterfactual scenarios to isolate the effect of racial bias features in training data.
• Utilized NLP models trained on racially biased datasets to assess their impact on user decision-making and persuasiveness.
• Measured changes in user behavior and decision quality when interacting with biased versus unbiased model outputs.
• Applied causal inference techniques to evaluate the direct effect of bias in data and models on user outcomes.
• Collected user responses and behavioral metrics to analyze shifts in persuasiveness and decision-making consistency.

Experimental results

Research questions

• RQ1 How does racial bias in training data affect the persuasiveness of users when relying on AI-generated outputs?
• RQ2 To what extent does the presence of racial bias in machine learning models alter individual decision-making processes?
• RQ3 What is the causal relationship between biased data, model behavior, and user trust in AI-assisted decisions?
• RQ4 How does model explainability mediate the impact of bias on user decision-making and articulateness?
• RQ5 In what ways do biased AI systems contribute to discriminatory outcomes in user-facing NLP applications?

Key findings

• Racial bias in training data significantly reduces user persuasiveness when relying on model-generated responses.
• Users exposed to biased model outputs demonstrated altered decision-making behavior, favoring discriminatory outcomes.
• The study found a measurable negative impact on user trustworthiness and decision quality due to biased system design.
• Biased models diminished the clarity and effectiveness of user articulation, especially in persuasive contexts.
• The findings support a causal link between biased data, model behavior, and downstream user-level discrimination.
• The research underscores the need for improved model explainability and fairness auditing in AI development pipelines.