[Paper Review] Catch Me If You Can: Identifying Fraudulent Physician Reviews with Large Language Models Using Generative Pre-Trained Transformers

Aishwarya Deep Shukla, Laksh Agarwal+5 more|arXiv (Cornell University)|Apr 19, 2023

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

This study proposes using large language models, particularly GPT-3 and GPT-4, to detect fraudulent physician reviews using a real-world dataset of 38,048 reviews. It demonstrates that GPT-3 outperforms traditional machine learning models, requires less training data, and excels in cold-start scenarios, revealing that fake reviews are more clinically detailed, more reserved in sentiment, and better structured than genuine ones.

ABSTRACT

The proliferation of fake reviews of doctors has potentially detrimental consequences for patient well-being and has prompted concern among consumer protection groups and regulatory bodies. Yet despite significant advancements in the fields of machine learning and natural language processing, there remains limited comprehension of the characteristics differentiating fraudulent from authentic reviews. This study utilizes a novel pre-labeled dataset of 38048 physician reviews to establish the effectiveness of large language models in classifying reviews. Specifically, we compare the performance of traditional ML models, such as logistic regression and support vector machines, to generative pre-trained transformer models. Furthermore, we use GPT4, the newest model in the GPT family, to uncover the key dimensions along which fake and genuine physician reviews differ. Our findings reveal significantly superior performance of GPT-3 over traditional ML models in this context. Additionally, our analysis suggests that GPT3 requires a smaller training sample than traditional models, suggesting its appropriateness for tasks with scarce training data. Moreover, the superiority of GPT3 performance increases in the cold start context i.e., when there are no prior reviews of a doctor. Finally, we employ GPT4 to reveal the crucial dimensions that distinguish fake physician reviews. In sharp contrast to previous findings in the literature that were obtained using simulated data, our findings from a real-world dataset show that fake reviews are generally more clinically detailed, more reserved in sentiment, and have better structure and grammar than authentic ones.

Motivation & Objective

• To address the growing problem of fraudulent physician reviews that harm patient well-being and undermine trust in healthcare systems.
• To investigate the distinguishing linguistic and structural characteristics between fake and genuine physician reviews using real-world data.
• To evaluate the effectiveness of large language models, especially GPT-3 and GPT-4, in classifying fraudulent reviews compared to traditional machine learning models.
• To examine the performance of these models under low-data (cold-start) conditions, where few or no prior reviews exist for a physician.
• To uncover the key dimensions that differentiate fake from authentic reviews using interpretability tools from GPT-4.

Proposed method

• The study uses a novel, pre-labeled dataset of 38,048 physician reviews, collected from real-world sources, to train and evaluate classification models.
• It compares traditional machine learning models—logistic regression and support vector machines—against generative pre-trained transformer models, particularly GPT-3.
• GPT-4 is employed to analyze and interpret the key linguistic and structural differences between fake and genuine reviews through zero-shot and few-shot prompting.
• Model performance is evaluated using standard classification metrics such as accuracy, precision, recall, and F1-score.
• The study conducts ablation studies to assess model performance under cold-start conditions, where no prior reviews exist for a physician.
• Interpretability analysis using GPT-4 identifies salient features such as clinical detail, sentiment tone, and grammatical structure as discriminative dimensions.

Experimental results

Research questions

• RQ1 How do large language models like GPT-3 compare to traditional machine learning models in detecting fraudulent physician reviews?
• RQ2 What are the key linguistic and structural features that distinguish fake from genuine physician reviews in real-world data?
• RQ3 Does the performance of GPT-3 improve in low-data (cold-start) scenarios where no prior reviews exist for a physician?
• RQ4 How do the characteristics of fake reviews differ from those of authentic reviews in terms of clinical detail, sentiment expression, and grammatical quality?
• RQ5 Can GPT-4 reliably identify and explain the dimensions that differentiate fraudulent from authentic physician reviews?

Key findings

• GPT-3 significantly outperforms traditional machine learning models such as logistic regression and support vector machines in classifying fraudulent physician reviews.
• GPT-3 achieves superior performance with substantially smaller training data requirements, making it suitable for low-resource settings.
• The performance advantage of GPT-3 is most pronounced in cold-start scenarios, where no prior reviews exist for a physician.
• Fake reviews are found to be more clinically detailed, more reserved in sentiment, and better structured with improved grammar compared to authentic reviews—contrary to prior assumptions based on simulated data.
• GPT-4 analysis confirms that clinical specificity, sentiment restraint, and syntactic quality are the primary discriminative dimensions between fake and genuine reviews.
• The study reveals that fake reviews are not less sophisticated but more professionally styled, challenging the common assumption that fraudulently written reviews are low-quality or ungrammatical.