[Paper Review] "I had a solid theory before but it's falling apart": Polarizing Effects of Algorithmic Transparency

Aaron Springer, Steve Whittaker|arXiv (Cornell University)|Nov 6, 2018

Ethics and Social Impacts of AI4 citations

TL;DR

This paper investigates how algorithmic transparency affects user trust in AI systems, revealing a paradox: transparency can polarize user perceptions—either boosting or undermining confidence—depending on whether users already hold a mental model of the system. The study shows that transparency causes users with pre-existing theories to focus on flaws, reducing trust, while those without such models benefit from increased clarity.

ABSTRACT

The rise of machine learning has brought closer scrutiny to intelligent systems, leading to calls for greater transparency and explainable algorithms. We explore the effects of transparency on user perceptions of a working intelligent system for emotion detection. In exploratory Study 1, we observed paradoxical effects of transparency which improves perceptions of system accuracy for some participants while reducing accuracy perceptions for others. In Study 2, we test this observation using mixed methods, showing that the apparent transparency paradox can be explained by a mismatch between participant expectations and system predictions. We qualitatively examine this process, indicating that transparency can undermine user confidence by causing users to fixate on flaws when they already have a model of system operation. In contrast transparency helps if users lack such a model. Finally, we revisit the notion of transparency and suggest design considerations for building safe and successful machine learning systems based on our insights.

Motivation & Objective

• To examine the psychological impact of algorithmic transparency on user perceptions of AI system accuracy.
• To investigate why transparency sometimes improves, and sometimes harms, user trust in machine learning systems.
• To identify the role of users' pre-existing mental models in shaping responses to system explanations.
• To explore the conditions under which transparency enhances or undermines user confidence in intelligent systems.
• To inform design principles for trustworthy and effective machine learning systems based on user cognition and expectations.

Proposed method

• Conducted an exploratory Study 1 to observe user reactions to transparency in an emotion detection system.
• Employed mixed-methods in Study 2, combining qualitative interviews with quantitative perception assessments.
• Analyzed user responses to identify mismatches between expectations and system predictions as a driver of perception shifts.
• Used qualitative coding to examine how users interpret and react to transparency, particularly focusing on fixation on flaws.
• Contrasted user behavior between those with and without pre-existing mental models of system operation.
• Re-evaluated the concept of transparency through a cognitive lens, emphasizing user mental models as a key variable.

Experimental results

Research questions

• RQ1 How does providing transparency about an AI system's decision-making process affect users' perceived accuracy of the system?
• RQ2 Why does transparency sometimes reduce user trust despite providing more information?
• RQ3 What role do users' pre-existing mental models of system behavior play in shaping their response to transparency?
• RQ4 Under what conditions does transparency improve user confidence versus erode it?
• RQ5 How can transparency be designed to minimize polarizing effects and enhance trust in machine learning systems?

Key findings

• Transparency led to paradoxical outcomes: some users perceived the system as more accurate, while others perceived it as less accurate.
• Users who already had a mental model of the system were more likely to focus on flaws when shown transparency, reducing their trust.
• Users without pre-existing mental models benefited from transparency, as it helped them form clearer expectations and increased confidence.
• The mismatch between user expectations and system predictions was a primary driver of negative reactions to transparency.
• Transparency can undermine confidence when users fixate on inconsistencies or limitations once they are made visible.
• The study suggests that transparency is not universally beneficial and must be tailored to users' cognitive states and mental models.