[Paper Review] LLMs May Perform MCQA by Selecting the Least Incorrect Option

Haochun Wang, Sendong Zhao+4 more|arXiv (Cornell University)|Feb 2, 2024

Expert finding and Q&A systems4 citations

TL;DR

This paper challenges the validity of Multiple Choice Question Answering (MCQA) as a benchmark for evaluating Large Language Models (LLMs), revealing that LLMs often select the 'least incorrect' option rather than truly understanding questions. Through systematic experiments across varied question formats, the authors identify Response Variability Syndrome (REVAS), where model performance fluctuates significantly with minor changes in answer options, undermining MCQA's reliability as a measure of true comprehension.

ABSTRACT

In the field of NLP, Large Language Models (LLMs) have markedly enhanced performance across a variety of tasks. However, the comprehensive evaluation of LLMs remains an inevitable challenge for the community. Recently, the adoption of Multiple Choice Question Answering (MCQA) as a benchmark for assessing LLMs has gained considerable traction. However, concerns regarding the robustness of this evaluative method persist. Building upon previous discussions on the issue of extit{variability}, we reveal an additional dimension of concern: LLMs may perform MCQA by selecting the least incorrect option rather than distinctly correct. This observation suggests that LLMs might regard multiple options as correct, which could undermine the reliability of MCQA as a metric for evaluating LLMs. To address this challenge, we introduce an enhanced dataset augmentation method for MCQA, termed MCQA+, to provide a more accurate reflection of the model performance, thereby highlighting the necessity for more sophisticated evaluation mechanisms in the assessment of LLM capabilities.

Motivation & Objective

• To investigate the reliability of MCQA as an evaluation benchmark for LLMs.
• To identify inconsistencies in LLM responses when the same question is rephrased or restructured.
• To examine whether LLMs truly understand question semantics or merely select the most plausible option among flawed choices.
• To explore whether incorporating negative (incorrect) answer options during training improves model robustness and consistency.
• To propose a refined evaluation framework that accounts for response variability and promotes deeper understanding over superficial pattern matching.

Proposed method

• The authors create multiple variants of MCQA instances: original, reordered answer options, and converted to true/false format.
• They evaluate multiple LLMs—including LLaMA2-13B, Baichuan2-13B, and ChatGPT 3.5—across these variants to measure response consistency.
• A controlled experiment is conducted using Baichuan2-7B-Base, training on positive (correct) and negative (incorrect) answer pairs to assess the impact of negative examples.
• Three training strategies are compared: SFT-only, SFT after pre-training on positive instances, and SFT after pre-training on both positive and negative instances.
• The evaluation includes both MCQA and true/false formats to test generalization across question structures.
• Statistical analysis of prediction shifts across configurations identifies the REVAS phenomenon, where model outputs vary significantly despite identical underlying questions.

Experimental results

Research questions

• RQ1 Does the performance of LLMs remain consistent when the same question is presented with different answer orderings or formats?
• RQ2 To what extent do LLMs rely on the relative plausibility of options rather than semantic understanding during MCQA?
• RQ3 Can the inclusion of negative (incorrect) answer options during pre-training or fine-tuning improve LLMs' response consistency and reduce REVAS?
• RQ4 Why do LLMs frequently misclassify questions when misleading options are introduced, even if they were correct initially?
• RQ5 Is MCQA a valid proxy for assessing true knowledge acquisition and reasoning in LLMs, or does it primarily measure pattern matching?

Key findings

• LLMs exhibit significant response variability—defined as REsponse Variability Syndrome (REVAS)—when the same question is reformulated or presented with reordered options, indicating inconsistent reasoning.
• In the few-shot setting, 39.23% to 57.99% of LLMs corrected their initial misjudgments when misleading options were replaced, showing that model predictions are highly sensitive to option content.
• Pre-training solely on positive (correct) answer instances led to a 15.59% drop in MCQA accuracy and a 16.83% drop in True/False accuracy, suggesting that positive-only learning harms generalization.
• Incorporating negative (incorrect) answer instances during pre-training improved True/False accuracy by 30.53% and MCQA accuracy by 8.38% compared to SFT-only, demonstrating the value of learning from incorrect options.
• The study confirms that LLMs often select the 'least incorrect' option rather than the correct one, implying they may not fully understand the semantics of questions.
• Empirical results show that high accuracy on standard MCQA benchmarks does not reflect genuine understanding, as performance drops significantly under minor perturbations in answer options.