[Paper Review] Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations

Yongshuo Zong, Tingyang Yu+3 more|arXiv (Cornell University)|Oct 2, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper reveals a critical vulnerability in large language models (LLMs) and vision-language models (VLLMs): they exhibit severe performance degradation when answer choices in multiple-choice question-answering (MCQA) tasks are randomly permuted. Despite high accuracy on standard benchmarks, models like Llama2-13B suffer up to a 33.89% drop in accuracy, falling below random chance, indicating a fundamental lack of invariance to input permutation, which undermines trust in their robustness.

ABSTRACT

Large language and vision-language models are rapidly being deployed in practice thanks to their impressive capabilities in instruction following, in-context learning, and so on. This raises an urgent need to carefully analyse their robustness so that stakeholders can understand if and when such models are trustworthy enough to be relied upon in any given application. In this paper, we highlight a specific vulnerability in popular models, namely permutation sensitivity in multiple-choice question answering (MCQA). Specifically, we show empirically that popular models are vulnerable to adversarial permutation in answer sets for multiple-choice prompting, which is surprising as models should ideally be as invariant to prompt permutation as humans are. These vulnerabilities persist across various model sizes, and exist in very recent language and vision-language models. Code is available at https://github.com/ys-zong/FoolyourVLLMs.

Motivation & Objective

• To investigate the robustness of LLMs and VLLMs in multiple-choice question-answering (MCQA) under simple input perturbations.
• To identify whether high benchmark performance masks underlying fragility in model behavior.
• To examine whether common mitigation strategies like majority voting or calibration can effectively address permutation-induced performance drops.
• To determine if the vulnerability stems from positional bias or a deeper systemic flaw in model reasoning.

Proposed method

• Systematically permute the order of answer choices in MCQA datasets while keeping the question and content unchanged.
• Evaluate model predictions across all permutations until a wrong prediction is made, measuring robustness as the number of permutations before failure.
• Conduct ablation studies by removing distractor options to test if reduced choice sets improve resilience.
• Compare model behavior across multiple LLMs and VLLMs on diverse MCQA benchmarks including MMLU, AGI-Eval, and ScienceQA.
• Use qualitative examples to illustrate how permutation alters model predictions despite identical semantic content.
• Test mitigation strategies such as majority voting across permutations and calibration techniques to assess their effectiveness.

Figure 1: Schematic Illustration of a MCQA permutation attack.

Experimental results

Research questions

• RQ1 To what extent do LLMs and VLLMs maintain accuracy when answer choices in MCQA are randomly permuted?
• RQ2 Does the performance degradation stem from positional bias or a deeper failure in model invariance to input order?
• RQ3 Can existing mitigation techniques like majority voting or calibration effectively restore robustness to permutation attacks?
• RQ4 How does the removal of distractor options affect model vulnerability to permutation attacks?
• RQ5 Is the observed fragility consistent across different model architectures, sizes, and modalities (text-only vs. vision-language)?

Key findings

• Llama2-13B experiences a 33.89% drop in accuracy on the MMLU benchmark after random permutation of answer choices, falling below random chance levels.
• The vulnerability persists across a wide range of LLMs and VLLMs, including recent models like LLaMA, LLaMA2, and Otter-Llama, indicating a systemic issue.
• Even after removing distractor options, model performance remains highly sensitive to permutation, contradicting the expectation that fewer choices should improve robustness.
• Positional bias alone cannot explain the strength of the attack, as performance degradation occurs regardless of answer position, indicating a deeper flaw in reasoning or attention mechanisms.
• Majority voting across permutations fails to restore original performance, despite incurring a k!-fold increase in computational cost.
• Calibration techniques such as calibrate-before-use do not effectively mitigate the vulnerability, underscoring the need for architectural or training-level solutions.

Figure 2: Summary of MCQA adversarial attack results for both LLMs and VLLMs. The values are average accuracy across all benchmarking datasets.