[Paper Review] Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT

Mingzhe Hu, Shaoyan Pan+2 more|arXiv (Cornell University)|Apr 11, 2023

Artificial Intelligence in Healthcare and EducationMedicine20 citations

TL;DR

This paper reviews how language models, including ChatGPT, are being used to enhance medical imaging tasks such as captioning, report generation, finding extraction, and visual question answering, and discusses potential benefits for clinical workflows.

ABSTRACT

In this paper, we aimed to provide a review and tutorial for researchers in the field of medical imaging using language models to improve their tasks at hand. We began by providing an overview of the history and concepts of language models, with a special focus on large language models. We then reviewed the current literature on how language models are being used to improve medical imaging, emphasizing different applications such as image captioning, report generation, report classification, finding extraction, visual question answering, interpretable diagnosis, and more for various modalities and organs. The ChatGPT was specially highlighted for researchers to explore more potential applications. We covered the potential benefits of accurate and efficient language models for medical imaging analysis, including improving clinical workflow efficiency, reducing diagnostic errors, and assisting healthcare professionals in providing timely and accurate diagnoses. Overall, our goal was to bridge the gap between language models and medical imaging and inspire new ideas and innovations in this exciting area of research. We hope that this review paper will serve as a useful resource for researchers in this field and encourage further exploration of the possibilities of language models in medical imaging.

Motivation & Objective

• Provide an overview of the history and concepts of language models, with emphasis on large language models.
• Review current literature on language models applied to medical imaging tasks across modalities and organs.
• Highlight ChatGPT as a focal point for exploring new applications in medical imaging.
• Discuss potential benefits for clinical workflow efficiency, diagnostic accuracy, and timely diagnoses.

Proposed method

• Survey historical development of language models from N-grams to large language models.
• Review literature on applications in medical imaging such as image captioning, report generation, report classification, finding extraction, visual question answering, and interpretable diagnosis.
• Highlight ChatGPT and discuss potential avenues for its use in medical imaging research.
• Synthesize insights to bridge gaps between language models and medical imaging practice.

Experimental results

Research questions

• RQ1 How are language models currently applied to medical imaging tasks (e.g., captioning, reporting, finding extraction, VQA, interpretation) across modalities and organs?
• RQ2 What potential benefits and limitations do language models offer for clinical workflow efficiency and diagnostic accuracy in medical imaging?
• RQ3 What special roles or applications does ChatGPT present for medical imaging research and practice?
• RQ4 How can researchers leverage language models to inspire new ideas and innovations in medical imaging?

Key findings

• Language models are being used to improve several medical imaging tasks including captioning, report generation, report classification, finding extraction, visual question answering, and interpretable diagnosis across various modalities and organs.
• ChatGPT is highlighted as a tool for researchers to explore more potential applications in medical imaging.
• Accurate and efficient language models have the potential to improve clinical workflow efficiency, reduce diagnostic errors, and support timely and accurate diagnoses for healthcare professionals.