[Paper Review] Don't Ignore Dual Logic Ability of LLMs while Privatizing: A Data-Intensive Analysis in Medical Domain

Yanrui Du, Sendong Zhao+5 more|arXiv (Cornell University)|Sep 8, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces the CALLA dataset to study how Large Language Models (LLMs) acquire interactive medical knowledge from Chinese medical literature. It identifies the 'fact-following response' phenomenon—where LLMs uncritically affirm facts in questions—and shows that instruction fine-tuning (IFT) data highly correlated with medical literature significantly enhances LLMs' ability to correctly apply pre-trained medical knowledge in dialogue, outperforming IFT data from other sources.

ABSTRACT

Extensive studies have been devoted to privatizing general-domain Large Language Models (LLMs) as Domain-Specific LLMs via feeding specific-domain data. However, these privatization efforts often ignored a critical aspect: Dual Logic Ability, which is a core reasoning ability for LLMs. The dual logic ability of LLMs ensures that they can maintain a consistent stance when confronted with both positive and negative statements about the same fact. Our study focuses on how the dual logic ability of LLMs is affected during the privatization process in the medical domain. We conduct several experiments to analyze the dual logic ability of LLMs by examining the consistency of the stance in responses to paired questions about the same fact. In our experiments, interestingly, we observed a significant decrease in the dual logic ability of existing LLMs after privatization. Besides, our results indicate that incorporating general domain dual logic data into LLMs not only enhances LLMs' dual logic ability but also further improves their accuracy. These findings underscore the importance of prioritizing LLMs' dual logic ability during the privatization process. Our study establishes a benchmark for future research aimed at exploring LLMs' dual logic ability during the privatization process and offers valuable guidance for privatization efforts in real-world applications.

Motivation & Objective

• To investigate how LLMs acquire and apply medical knowledge from Chinese medical literature in interactive dialogue settings.
• To identify and address the 'fact-following response' phenomenon, where LLMs uncritically affirm facts in questions regardless of correctness.
• To design an automatic framework for constructing high-quality instruction fine-tuning (IFT) data from medical literature to improve interactive knowledge utilization.
• To evaluate the impact of IFT data quality and source correlation on LLM performance in medical dialogue tasks.
• To explore real-world applications of combining literature-based and general-domain LLM responses for more comprehensive medical advice.

Proposed method

• The CALLA dataset is constructed with three components: a medical literature corpus, IFT data for supervised fine-tuning, and artificially generated test data with both consistent and inconsistent facts relative to the golden fact.
• A free-dialogue fact-checking task is used as the evaluation benchmark to probe LLMs' knowledge mastery without constraints from instructions or demonstrations.
• An automatic IFT data construction framework is proposed, leveraging LLMs (e.g., ChatGPT) to generate dialogues enriched with medical literature content, improving information integration over direct prompting.
• The framework includes a data cleaning module to convert subjective expressions in responses into objective ones, enhancing reliability for real-world use.
• A score based on the Longest Common Substring (LCS) between responses and medical literature is used to quantify the degree of literature information integration in IFT data.
• LoRA-based parameter-efficient fine-tuning is applied due to GPU memory constraints, enabling efficient adaptation of base LLMs on the CALLA dataset.

Experimental results

Research questions

• RQ1 How well can LLMs acquire and apply medical knowledge from Chinese medical literature in interactive dialogue settings?
• RQ2 To what extent does the 'fact-following response' phenomenon distort the evaluation of LLMs' medical knowledge mastery?
• RQ3 How does the correlation between IFT data and medical literature affect LLMs' performance in interactive medical knowledge tasks?
• RQ4 Can an automatic framework effectively generate high-quality IFT data from medical literature that enhances LLMs' interactive knowledge utilization?
• RQ5 How can literature-based and general-domain LLM responses be combined to produce more comprehensive and practical medical advice?

Key findings

• The 'fact-following response' phenomenon causes LLMs to affirm incorrect facts in questions, leading to inaccurate evaluation when only consistent test cases are used.
• Using both consistent and inconsistent test cases reveals that LLMs often fail to reject false facts, highlighting the need for robust evaluation protocols.
• IFT data highly correlated with medical literature improves LLMs' ability to apply pre-trained medical knowledge in dialogue, increasing accuracy in fact-checking tasks.
• The proposed automatic IFT data construction framework increases literature information integration by 18.8 percentage points (65.86% vs. 47.06%) compared to direct prompting.
• Data cleaning reduces the occurrence of subjective expressions in LLM responses from 44% to 7%, significantly improving response objectivity for real-world applications.
• Combining responses from literature-based and general-domain LLMs yields more comprehensive medical advice than either model alone.