[Paper Review] Leveraging Large Language Models for Multiple Choice Question Answering

Joshua A. Robinson, Christopher Michael Rytting+1 more|arXiv (Cornell University)|Oct 22, 2022

Topic Modeling40 citations

TL;DR

The paper shows that prompting LLMs with multiple-choice prompts (MCP)—where answer options are bound to symbols and presented together with the question—greatly enhances MCQA performance, often surpassing cloze prompting (CP) and approaching or beating SOTA on many datasets, especially for models with strong multiple-choice symbol binding (MCSB) ability.

ABSTRACT

While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., "A") associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated.

Motivation & Objective

• Motivate and formalize the limitations of cloze prompting for MCQA in large language models.
• Propose multiple-choice prompting (MCP) as an alternative that binds answer options to symbols and prompts the model to choose among them.
• Introduce and formalize multiple-choice symbol binding (MCSB) as a key capability for effective MCP.
• Empirically evaluate MCP across 20 diverse MCQA datasets using models with varying MCSB ability, and compare to SOTA.

Proposed method

• Define and contrast cloze prompting (CP) and multiple-choice prompting (MCP).
• Introduce multiple-choice symbol binding (MCSB) as the ability to consistently associate answer options with symbols despite answer order.
• Evaluate several LLMs (GPT-3, Codex, InstructGPT, etc.) for MCSB using PPA (Proportion of Plurality Agreement) on OpenBookQA.
• Conduct large-scale experiments across 20 MCQA datasets comparing CP (with various normalizations) vs MCP.
• Use Codex (Davinci) as the primary evaluation model due to high MCSB ability and cost considerations.
• Report zero-, one-, and few-shot performance under CP and MCP, highlighting gains from MCP.

Experimental results

Research questions

• RQ1 Does multiple-choice prompting (MCP) enable LLMs to better utilize answer options compared to cloze prompting (CP)?
• RQ2 How does a model's multiple-choice symbol binding (MCSB) ability affect MCQA performance with MCP across diverse datasets?
• RQ3 To what extent can MCP close the gap to SOTA on a broad suite of MCQA tasks without task-specific tuning?

Key findings

• MCP outperforms CP on 16 of 20 datasets; the average improvement over CP is 8.3–12.2 percentage points across zero-, one-, and few-shot settings.
• Across the 20 datasets, MCP yields a mean gap of 9.7 percentage points over CP and a maximum gap of 44% in some tasks.
• MCP surpasses old SOTA scores on 9 of 20 datasets, with gains up to 15% on a single task, and MCP scores average within 0.6% of SOTA across datasets.
• Codex and Instruct models with high MCSB ability achieve the strongest gains from MCP, often approaching or beating SOTA without task-specific tuning.
• MCP achieves comparable or better performance with 4.3x fewer forward passes (API calls) than the best CP strategies across tasks and exemplar settings.