[Paper Review] Large Language Models Streamline Automated Machine Learning for Clinical Studies

Soroosh Tayebi Arasteh, Tianyu Han+5 more|arXiv (Cornell University)|Aug 27, 2023

Machine Learning in Healthcare6 citations

TL;DR

This study evaluates ChatGPT Advanced Data Analysis (ADA), a large language model (LLM)-powered tool, for autonomously developing machine learning models in clinical research. Without specialized training, ChatGPT ADA generated state-of-the-art models matching or exceeding manually crafted benchmarks across four real-world clinical trials, demonstrating high accuracy and reliability in predicting outcomes like cancer progression and biomarkers.

ABSTRACT

A knowledge gap persists between machine learning (ML) developers (e.g., data scientists) and practitioners (e.g., clinicians), hampering the full utilization of ML for clinical data analysis. We investigated the potential of the ChatGPT Advanced Data Analysis (ADA), an extension of GPT-4, to bridge this gap and perform ML analyses efficiently. Real-world clinical datasets and study details from large trials across various medical specialties were presented to ChatGPT ADA without specific guidance. ChatGPT ADA autonomously developed state-of-the-art ML models based on the original study's training data to predict clinical outcomes such as cancer development, cancer progression, disease complications, or biomarkers such as pathogenic gene sequences. Following the re-implementation and optimization of the published models, the head-to-head comparison of the ChatGPT ADA-crafted ML models and their respective manually crafted counterparts revealed no significant differences in traditional performance metrics (P>0.071). Strikingly, the ChatGPT ADA-crafted ML models often outperformed their counterparts. In conclusion, ChatGPT ADA offers a promising avenue to democratize ML in medicine by simplifying complex data analyses, yet should enhance, not replace, specialized training and resources, to promote broader applications in medical research and practice.

Motivation & Objective

• To assess whether large language models like ChatGPT ADA can autonomously develop high-performing machine learning models for clinical data without requiring specialized ML training.
• To evaluate the reliability and validity of LLM-generated models compared to manually crafted, expert-recommended models in real-world clinical trial datasets.
• To investigate whether non-expert clinicians can effectively use LLMs to perform complex ML analyses in medical research.
• To determine whether LLMs can streamline automated machine learning (AutoML) in clinical settings while maintaining model performance and interpretability.

Proposed method

• Real-world clinical datasets from four large, multi-specialty trials were input into ChatGPT ADA with minimal prompting (e.g., 'Analyze this patient data and build a machine learning model predicting 12-month mortality').
• ChatGPT ADA autonomously selected and implemented machine learning models—primarily Random Forest and LightGBM—based on data characteristics and task requirements.
• The LLM generated Python code for data preprocessing, model training, hyperparameter selection, and evaluation, including model interpretation and result formatting.
• Model performance was evaluated using standard metrics (e.g., AUC, accuracy) and compared to both the original published models and re-implemented versions by expert data scientists.
• Hyperparameter details, validation strategies, and data scaling decisions were extracted from the LLM’s internal reasoning and responses to assess methodological rigor.
• A comparative analysis was conducted between LLM-generated models and manually re-implemented models to assess statistical equivalence and performance differences.

Experimental results

Research questions

• RQ1 Can ChatGPT ADA autonomously generate clinically relevant machine learning models from raw clinical trial data without expert guidance?
• RQ2 How do the performance metrics of LLM-generated models compare to those of manually crafted, expert-recommended models in real-world clinical datasets?
• RQ3 Does the LLM’s model selection and hyperparameter configuration align with best practices in machine learning, particularly regarding validation and regularization?
• RQ4 To what extent can non-technical clinicians use LLMs like ChatGPT ADA to perform advanced data analysis in medical research?
• RQ5 What are the limitations and risks of relying on LLMs for model development in high-stakes clinical applications?

Key findings

• ChatGPT ADA successfully generated machine learning models for four distinct clinical trials across different specialties, including cancer progression, disease complications, and biomarker prediction.
• The performance of LLM-generated models showed no statistically significant difference compared to manually re-implemented models, with p-values ≥ 0.072 across all comparisons.
• In multiple cases, the LLM-generated models outperformed their manually crafted counterparts in terms of AUC and accuracy, suggesting potential for superior generalization.
• The LLM selected appropriate algorithms (e.g., Random Forest, LightGBM) and justified hyperparameter choices, such as n_estimators=1000 and min_samples_leaf=1, with clear reasoning.
• ChatGPT ADA correctly identified that data scaling was unnecessary for tree-based models and explained the underlying rationale, demonstrating understanding of model-specific requirements.
• The LLM acknowledged limitations such as the absence of grid search or explicit validation splits in the initial run, indicating awareness of best practices and potential improvements.