[論文レビュー] Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today

Zhuo Wang, Rongzhen Li+9 more|arXiv (Cornell University)|Jun 2, 2023

Topic Modeling被引用数 9

ひとこと要約

The paper evaluates GPT-4 versus traditional AI tools and doctors on two real clinical dementia datasets, finding GPT-4 does not yet outperform traditional AI models in dementia diagnosis, though it shows potential under certain settings.

ABSTRACT

Recent investigations show that large language models (LLMs), specifically GPT-4, not only have remarkable capabilities in common Natural Language Processing (NLP) tasks but also exhibit human-level performance on various professional and academic benchmarks. However, whether GPT-4 can be directly used in practical applications and replace traditional artificial intelligence (AI) tools in specialized domains requires further experimental validation. In this paper, we explore the potential of LLMs such as GPT-4 to outperform traditional AI tools in dementia diagnosis. Comprehensive comparisons between GPT-4 and traditional AI tools are conducted to examine their diagnostic accuracy in a clinical setting. Experimental results on two real clinical datasets show that, although LLMs like GPT-4 demonstrate potential for future advancements in dementia diagnosis, they currently do not surpass the performance of traditional AI tools. The interpretability and faithfulness of GPT-4 are also evaluated by comparison with real doctors. We discuss the limitations of GPT-4 in its current state and propose future research directions to enhance GPT-4 in dementia diagnosis.

研究の動機と目的

• Assess whether GPT-4 can replace traditional AI tools in dementia diagnosis in real clinical settings.
• Compare GPT-4 and GPT-3.5 with interpretable and black-box baselines on two real datasets (ADNI and PUMCH).
• Evaluate interpretability and faithfulness of GPT-4 against physician judgments.
• Investigate limitations of GPT-4 and propose directions to enhance its role in dementia diagnosis.

提案手法

• Design simple GPT-4 prompt templates that convert dementia diagnosis into a multiple-choice task with features such as demographics, cognitive test results, and biomarkers.
• Conduct a 90/10 train/test split on two real clinical datasets (ADNI and private PUMCH) to assess diagnostic accuracy.
• Compare GPT-4 and GPT-3.5 against five traditional models (CART, Logistic Regression, RRL, Random Forest, XGBoost) across binary and ternary classification tasks.
• Evaluate GPT-4’s interpretability and faithfulness by qualitative comparisons with doctor diagnoses on the PUMCH dataset.
• Discuss information leakage concerns by using a private dataset to benchmark GPT-4.
• Analyze factors affecting GPT-4 performance, such as input quality and tabular data handling.

実験結果

リサーチクエスチョン

• RQ1 Can GPT-4 achieve dementia diagnostic accuracy comparable to traditional AI tools on real clinical data?
• RQ2 How does GPT-4 compare to GPT-3.5 and to interpretable/black-box traditional models in dementia diagnosis?
• RQ3 What are the interpretability and faithfulness characteristics of GPT-4 relative to doctors?
• RQ4 What limitations prevent GPT-4 from matching or exceeding traditional AI tools in this domain, and how can future work address them?

主な発見

• GPT-4 and GPT-3.5 do not currently outperform traditional AI tools like RRL in dementia diagnosis across the evaluated datasets.
• GPT-4 shows better performance than GPT-3.5 in some cases (notably on the ADNI dataset) but remains behind RRL on PUMCH-T and overall,
• few-shot prompting can improve GPT-4 performance in some scenarios.
• private datasets were used to mitigate information leakage and yield more realistic benchmarks.
• GPT-4 provides readable explanations and can align with doctor diagnostics in some cases but may misdiagnose input-sensitive cases and lacks guaranteed faithfulness.