[Paper Review] How Susceptible are LLMs to Influence in Prompts?

Sotiris Anagnostidis, Jannis Bulian|arXiv (Cornell University)|Aug 17, 2024

Artificial Intelligence in Law4 citations

TL;DR

This study investigates how susceptible large language models (LLMs) are to influence from augmented inputs, such as predictions and explanations from other models. Using Llama2, Mixtral, and Falcon across diverse question-answering tasks, it finds that LLMs are strongly swayed by external inputs—even when explanations are factually incorrect—especially when the input is framed as authoritative or highly confident, highlighting risks in retrieval-augmented and multi-agent systems.

ABSTRACT

Large Language Models (LLMs) are highly sensitive to prompts, including additional context provided therein. As LLMs grow in capability, understanding their prompt-sensitivity becomes increasingly crucial for ensuring reliable and robust performance, particularly since evaluating these models becomes more challenging. In this work, we investigate how current models (Llama, Mixtral, Falcon) respond when presented with additional input from another model, mimicking a scenario where a more capable model -- or a system with access to more external information -- provides supplementary information to the target model. Across a diverse spectrum of question-answering tasks, we study how an LLM's response to multiple-choice questions changes when the prompt includes a prediction and explanation from another model. Specifically, we explore the influence of the presence of an explanation, the stated authoritativeness of the source, and the stated confidence of the supplementary input. Our findings reveal that models are strongly influenced, and when explanations are provided they are swayed irrespective of the quality of the explanation. The models are more likely to be swayed if the input is presented as being authoritative or confident, but the effect is small in size. This study underscores the significant prompt-sensitivity of LLMs and highlights the potential risks of incorporating outputs from external sources without thorough scrutiny and further validation. As LLMs continue to advance, understanding and mitigating such sensitivities will be crucial for their reliable and trustworthy deployment.

Motivation & Objective

• To understand how LLMs respond to augmented inputs from other models in question-answering tasks.
• To investigate whether the quality, authoritativeness, or confidence level of external inputs affects LLM responses.
• To assess whether prompting strategies can mitigate undue influence from external sources.
• To evaluate the extent of prompt-sensitivity in state-of-the-art open-source LLMs across diverse reasoning and knowledge tasks.
• To highlight risks of error propagation and bias reinforcement when integrating external model outputs without validation.

Proposed method

• Conducted experiments using three open-source LLMs: Llama2, Mixtral, and Falcon.
• Evaluated model responses on eight diverse question-answering benchmarks: PIQA, SIQA, CommonsenseQA, OpenBookQA, WikiQA, GPQA, QuALITY, and BoolQ.
• Introduced an external model (advocate) that provides a predicted answer and reasoning to the target model (judge) in the prompt.
• Varied three key influence factors: presence of explanation, source authoritativeness (5 levels), and stated confidence level.
• Measured changes in the judge model’s response when exposed to advocate inputs versus baseline prompts.
• Used controlled prompting to assess whether mitigation strategies could reduce susceptibility.

Figure 1: Leveraging external evidence, augmented inputs can help LLMs provide more informed answers.

Experimental results

Research questions

• RQ1 How does the inclusion of an external model’s explanation affect the target LLM’s response to multiple-choice questions?
• RQ2 To what extent does the perceived authority of the external source influence the target LLM’s decision-making?
• RQ3 Does the stated confidence of the external input amplify or reduce the target model’s susceptibility?
• RQ4 Can prompting strategies effectively reduce the influence of external inputs on LLM outputs?
• RQ5 How does the correctness of the explanation impact the target model’s response, especially when the explanation is factually wrong?

Key findings

• LLMs are strongly influenced by external model inputs, even when the provided explanation is factually incorrect.
• The presence of an explanation increases susceptibility, regardless of its quality or logical validity.
• Models are more likely to adopt the advocated answer when the input is labeled as authoritative or highly confident, though the effect size is small.
• The influence increases with higher levels of perceived authoritativeness and stated confidence, indicating sensitivity to source credibility cues.
• Even highly confident baseline responses from the judge model are not fully protected from external influence.
• Standard prompting techniques are insufficient to mitigate this susceptibility, indicating a need for deeper architectural or reasoning-level solutions.

Figure 3: Influence of advocates’ responses to judges’ predictions. Shading indicates whether an advocate provides their argument why their choice is the correct one, as seen in Tab. 2 (bottom right).