[Paper Review] A study on the adequacy of common IQA measures for medical images

Anna Breger, Clemens Karner+12 more|arXiv (Cornell University)|May 29, 2024

Consumer Perception and Purchasing BehaviorBusiness, Management and Accounting3 citations

TL;DR

This study evaluates the suitability of common image quality assessment (IQA) measures for medical images using expert ratings on chest X-rays and photoacoustic images, alongside supplementary data on natural images and accelerated MRI. Results show PSNR and SSIM perform poorly in medical contexts, while HaarPSI, DISTS, FSIM, and LPIPS demonstrate superior correlation with human ratings, indicating the need for task-specific IQA metrics in medical imaging.

ABSTRACT

Image quality assessment (IQA) is standard practice in the development stage of novel machine learning algorithms that operate on images. The most commonly used IQA measures have been developed and tested for natural images, but not in the medical setting. Reported inconsistencies arising in medical images are not surprising, as they have different properties than natural images. In this study, we test the applicability of common IQA measures for medical image data by comparing their assessment to manually rated chest X-ray (5 experts) and photoacoustic image data (2 experts). Moreover, we include supplementary studies on grayscale natural images and accelerated brain MRI data. The results of all experiments show a similar outcome in line with previous findings for medical images: PSNR and SSIM in the default setting are in the lower range of the result list and HaarPSI outperforms the other tested measures in the overall performance. Also among the top performers in our experiments are the full reference measures FSIM, LPIPS and MS-SSIM. Generally, the results on natural images yield considerably higher correlations, suggesting that additional employment of tailored IQA measures for medical imaging algorithms is needed.

Motivation & Objective

• To assess the adequacy of widely used IQA measures—originally designed for natural images—on medical imaging data.
• To address the lack of expert-annotated, publicly available datasets for reliable IQA benchmarking in medical imaging.
• To investigate whether common full-reference (FR) and no-reference (NR) IQA metrics correlate effectively with human expert perception in clinical image quality assessment.
• To identify robust, generalizable IQA measures suitable for medical image restoration and reconstruction tasks.
• To highlight the limitations of PSNR and SSIM in medical contexts and advocate for tailored or improved metrics in clinical AI development.

Proposed method

• Conducted four independent experiments using manually rated medical image datasets: chest X-rays (5 experts), photoacoustic images (1 expert), grayscale natural images (5 raters), and accelerated brain MRI reconstructions.
• Employed standardized IQA metrics including PSNR, SSIM, MS-SSIM, DISTS, LPIPS, HaarPSI, and NR measures like NIQE and PaQ-2-PIQ.
• Used the speedyIQA annotation tool to collect and standardize expert ratings, applying z-score normalization across graders before correlation analysis.
• Evaluated performance using Spearman and Kendall rank correlation coefficients (SRCC/KRCC) between IQA scores and manual ratings.
• Utilized official implementations of IQA measures, with PyTorch Lightning used for VIF due to unavailability of original code.
• Tested generalizability across image types (grayscale, color), modalities (X-ray, photoacoustic, MRI), and distortion types (noise, compression, acceleration).

Experimental results

Research questions

• RQ1 How well do commonly used IQA measures correlate with expert ratings on medical images such as chest X-rays and photoacoustic scans?
• RQ2 Do standard IQA metrics like PSNR and SSIM remain reliable for assessing medical image quality, especially when compared to human perception?
• RQ3 Which IQA measures show the highest generalizability and robustness across diverse medical imaging tasks and modalities?
• RQ4 How do full-reference and no-reference IQA measures compare in their ability to reflect human-annotated quality rankings in medical imaging?
• RQ5 To what extent do IQA metrics trained on natural images generalize to medical image data with distinct characteristics (e.g., grayscale, low contrast, high diagnostic detail)?

Key findings

• PSNR and SSIM in default settings showed the lowest correlation with expert ratings across all medical image datasets, confirming their limited suitability for medical image quality assessment.
• HaarPSI consistently outperformed other IQA measures, ranking among the top three in all tested medical imaging tasks and demonstrating strong generalizability.
• Full-reference measures such as DISTS, FSIM, LPIPS, and MS-SSIM showed strong performance across multiple datasets, indicating their potential for use in medical image evaluation.
• Correlation coefficients for IQA measures were significantly higher on grayscale natural image datasets than on medical image datasets, highlighting a performance gap in medical contexts.
• NR measures like NIQE and PaQ-2-PIQ struggled to detect quality degradation in accelerated MRI reconstructions, indicating poor sensitivity to clinically relevant distortions.
• The study confirms the need for task-specific and modality-aware IQA metrics in medical imaging, as general-purpose metrics like PSNR and SSIM fail to align with expert perception.