[Paper Review] Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness

Guido Zuccon, Bevan Koopman|arXiv (Cornell University)|Feb 23, 2023

Topic ModelingComputer Science33 citations

TL;DR

The paper compares ChatGPT's health question answering using only model knowledge versus using prompt-provided evidence, showing that prompt knowledge can overturn model knowledge and reduce accuracy from 80% to 63%.

ABSTRACT

Generative pre-trained language models (GPLMs) like ChatGPT encode in the model's parameters knowledge the models observe during the pre-training phase. This knowledge is then used at inference to address the task specified by the user in their prompt. For example, for the question-answering task, the GPLMs leverage the knowledge and linguistic patterns learned at training to produce an answer to a user question. Aside from the knowledge encoded in the model itself, answers produced by GPLMs can also leverage knowledge provided in the prompts. For example, a GPLM can be integrated into a retrieve-then-generate paradigm where a search engine is used to retrieve documents relevant to the question; the content of the documents is then transferred to the GPLM via the prompt. In this paper we study the differences in answer correctness generated by ChatGPT when leveraging the model's knowledge alone vs. in combination with the prompt knowledge. We study this in the context of consumers seeking health advice from the model. Aside from measuring the effectiveness of ChatGPT in this context, we show that the knowledge passed in the prompt can overturn the knowledge encoded in the model and this is, in our experiments, to the detriment of answer correctness. This work has important implications for the development of more robust and transparent question-answering systems based on generative pre-trained language models.

Motivation & Objective

• Assess ChatGPT's effectiveness at answering complex health information questions using only model knowledge (question-only)
• Evaluate how prompting with supporting or contrary evidence influences answer correctness (evidence-biased)
• Determine how prompt-embedded knowledge impacts health information reliability and potential misinformation risk

Proposed method

• Use 100 topics from the TREC Health Misinformation track to test general effectiveness (RQ1)
• Compare question-only prompts to evidence-biased prompts with up to 3 supportive and 3 contrary documents per topic (RQ2)
• Annotate and evaluate ChatGPT responses against ground truth using Yes/No plus explanations
• Analyze how often evidence-biased prompts flip correct answers and whether flips improve or worsen accuracy

Experimental results

Research questions

• RQ1 RQ1 General Effectiveness: How effective is ChatGPT at answering complex health information questions?
• RQ2 RQ2 Evidence Biased Effectiveness: How does prompting with supporting or contrary evidence influence answer correctness?

Key findings

• ChatGPT achieves 80% accuracy when answering health questions using knowledge encoded in the model alone.
• With evidence-biased prompting, overall accuracy drops to 63%.
• Prompt-provided evidence can overturn the model's answer, often leading to incorrect results when evidence is contrary.
• Flips in answer due to evidence-biased prompts tend to be incorrect more often than not.
• Explanations accompanying answers frequently discuss limited or conflicting evidence and sometimes general medical advice, not always verifiable sources.