[Paper Review] Can Large Language Models Challenge CNNs in Medical Image Analysis?

Shibbir Ahmed, Shahnewaz Karim Sakib+1 more|ArXiv.org|May 29, 2025

Radiomics and Machine Learning in Medical Imaging3 citations

TL;DR

The paper compares CNNs and LLMs (GPT-4o and Llama3.2-vision) for medical image classification across X-ray, MRI, and CT datasets, showing CNNs generally outperform while enhanced data filtering improves LLM performance and efficiency.

ABSTRACT

This study presents a multimodal AI framework designed for precisely classifying medical diagnostic images. Utilizing publicly available datasets, the proposed system compares the strengths of convolutional neural networks (CNNs) and different large language models (LLMs). This in-depth comparative analysis highlights key differences in diagnostic performance, execution efficiency, and environmental impacts. Model evaluation was based on accuracy, F1-score, average execution time, average energy consumption, and estimated $CO_2$ emission. The findings indicate that although CNN-based models can outperform various multimodal techniques that incorporate both images and contextual information, applying additional filtering on top of LLMs can lead to substantial performance gains. These findings highlight the transformative potential of multimodal AI systems to enhance the reliability, efficiency, and scalability of medical diagnostics in clinical settings.

Motivation & Objective

• Benchmark CNNs and LLMs on diverse medical imaging datasets (X-ray, MRI, CT).
• Analyze computational efficiency and environmental impact of CNNs and LLMs.
• Investigate enhanced data filtering to improve LLM classification reliability and efficiency.
• Assess confidence calibration and reliability of model predictions in clinical settings.

Proposed method

• Evaluate CNN and LLM architectures (GPT-4o, Llama3.2-vision) on chest X-ray, brain MRI, and chest CT datasets.
• Use accuracy, F1-score, and average confidence score as performance metrics.
• Measure average execution time, energy consumption, and estimated CO2 emissions for resource efficiency.
• Implement a multi-stage data filtering pipeline to improve LLM classification via context features and targeted questions.

Fig. 3 : Context Aggregation and Question Formulation

Experimental results

Research questions

• RQ1 How do CNNs and LLMs compare in accuracy and F1-score across different medical imaging modalities?
• RQ2 What are the trade-offs between predictive performance and computational resource usage for CNNs vs. LLMs?
• RQ3 Can enhanced data filtering improve LLM diagnostic performance and efficiency in medical imaging tasks?
• RQ4 How reliable are LLM predictions as indicated by confidence scores compared to CNNs?

Key findings

• CNNs outperform LLMs on all three modalities in accuracy and F1-score (Chest X-ray: CNN 0.83 vs. GPT-4o 0.62; MRI: CNN 0.98+ vs. GPT-4o 0.60; Chest CT: CNN 0.91 vs. GPT-4o 0.22).
• GPT-4o achieves higher average confidence scores than CNNs but with much lower accuracy, indicating overconfidence in errors.
• Llama3.2-vision shows moderate accuracy (Chest X-ray 0.65; MRI 0.52; Chest CT 0.50) and relatively high confidence scores, still below CNN performance.
• Enhanced data filtering for LLMs (with context questions) raises Chest X-ray accuracy from 62% to 82.01% and reduces average execution time from 6.23 s to 2.35 s and energy from 1.84 to 1.65 W-H.
• CNNs exhibit the best computational efficiency (lowest execution time, energy use, and CO2 emissions) across datasets; LLMs incur higher costs, especially Llama3.2-vision.
• Table 2 shows significant gains from filtering for GPT-4o (accuracy 62% w/o vs 82.01% with filtering; time 6.23 s vs 2.35 s; energy 1.84 W-H vs 1.65 W-H).