[Paper Review] "AI enhances our performance, I have no doubt this one will do the same": The Placebo effect is robust to negative descriptions of AI

Agnes Mercedes Kloft, Robin Welsch+2 more|arXiv (Cornell University)|Sep 28, 2023

Pain Management and Placebo EffectNeuroscience3 citations

TL;DR

This study demonstrates that placebo effects from AI are highly robust, as participants performed better and processed information faster when believing a sham AI was active—even when explicitly told the AI would impair performance. Using Bayesian cognitive modeling, the authors show that expectations, not verbal descriptions, drive behavioral improvements, revealing a pervasive AI performance bias that undermines traditional expectation controls in HCI research.

ABSTRACT

Heightened AI expectations facilitate performance in human-AI interactions through placebo effects. While lowering expectations to control for placebo effects is advisable, overly negative expectations could induce nocebo effects. In a letter discrimination task, we informed participants that an AI would either increase or decrease their performance by adapting the interface, but in reality, no AI was present in any condition. A Bayesian analysis showed that participants had high expectations and performed descriptively better irrespective of the AI description when a sham-AI was present. Using cognitive modeling, we could trace this advantage back to participants gathering more information. A replication study verified that negative AI descriptions do not alter expectations, suggesting that performance expectations with AI are biased and robust to negative verbal descriptions. We discuss the impact of user expectations on AI interactions and evaluation and provide a behavioral placebo marker for human-AI interaction

Motivation & Objective

• To investigate whether negative verbal descriptions of AI can suppress placebo effects in human-AI interaction.
• To examine how user expectations influence decision-making and performance in the presence of a sham AI system.
• To determine whether nocebo effects (negative expectations) can be induced through verbal framing in AI interaction studies.
• To assess the robustness of placebo effects in AI by comparing behavioral, subjective, and cognitive modeling outcomes.
• To propose strategies for mitigating expectation bias in HCI evaluations of AI technologies.

Proposed method

• Conducted an experimental study with 66 participants using a letter discrimination task to measure decision-making under sham-AI conditions.
• Used Bayesian cognitive modeling of the drift-diffusion model (DDM) to estimate changes in drift rate (ν) and non-decision time (τ) as indicators of information processing speed and response style.
• Manipulated verbal descriptions of AI as either positive or negative to assess their impact on expectations and performance.
• Replicated findings in an online study with 95 participants to verify robustness of the AI performance bias.
• Applied a placebo-controlled design with a non-functional AI interface to isolate the effect of expectations from actual system functionality.
• Used subjective performance ratings and behavioral data to triangulate findings across multiple levels of analysis.

Figure 1 . Schematic representation of the drift-diffusion process of decision-making with an increased drift-rate $\nu$ and a decreased non-decision time $\tau$ , for the sham-AI active (sAI active) condition as compared to the sham-AI inactive (sAI inactive) condition. When using a sham AI, partic

Experimental results

Research questions

• RQ1 Does a negative verbal description of an AI system reduce performance expectations and lead to a nocebo effect?
• RQ2 Can placebo effects from a sham AI be observed at the behavioral level, even when expectations are explicitly lowered?
• RQ3 To what extent do cognitive processing parameters (e.g., drift rate, non-decision time) change under placebo conditions?
• RQ4 Is the AI performance bias—where users expect improvement regardless of description—robust across different study formats?
• RQ5 How can expectation bias in AI evaluation be mitigated in future HCI research?

Key findings

• Participants showed a significant behavioral placebo effect, gathering information faster (increased drift rate ν) and responding more quickly (reduced non-decision time τ) when believing a sham AI was active, regardless of verbal description.
• Despite being told the AI would impair performance, participants maintained high expectations for AI effectiveness, indicating a robust AI performance bias.
• The Bayesian cognitive model revealed that the placebo effect was driven by faster information accumulation, not changes in response caution or threshold.
• A replication study with 95 participants confirmed the persistence of the AI performance bias, even under negative framing.
• The study found no evidence of a nocebo effect, as negative descriptions failed to reduce performance expectations or impair behavior.
• Subjective performance ratings showed a medium effect size (dz = 0.53), while behavioral effects were small (dz = 0.12), suggesting that expectations influence perception more than behavior, though both are affected.

Figure 2 . Trial sequence during the letter discrimination task. The duration of the ISI followed a Gaussian distribution ( M = 1000 ms, SD = 600 ms). Key responses (left or right arrow) were logged during the presentation of the mask.