[Paper Review] The Power Of Simplicity: Why Simple Linear Models Outperform Complex Machine Learning Techniques -- Case Of Breast Cancer Diagnosis

Muhammad Arshad, Sakib Shahriar+1 more|arXiv (Cornell University)|Jun 4, 2023

Artificial Intelligence in HealthcareHealth Professions3 citations

TL;DR

This study demonstrates that simple linear models, particularly Logistic Regression (LR), outperform complex machine learning techniques like Decision Trees (DT) and Support Vector Machines (SVM) in breast cancer diagnosis using the UCI Breast Cancer Wisconsin dataset. LR achieved a test accuracy of 97.28% with high interpretability and computational efficiency (35.56 ms), outperforming DT (93.73%) and SVM (96.44%), highlighting the advantages of simplicity in medical AI applications.

ABSTRACT

This research paper investigates the effectiveness of simple linear models versus complex machine learning techniques in breast cancer diagnosis, emphasizing the importance of interpretability and computational efficiency in the medical domain. We focus on Logistic Regression (LR), Decision Trees (DT), and Support Vector Machines (SVM) and optimize their performance using the UCI Machine Learning Repository dataset. Our findings demonstrate that the simpler linear model, LR, outperforms the more complex DT and SVM techniques, with a test score mean of 97.28%, a standard deviation of 1.62%, and a computation time of 35.56 ms. In comparison, DT achieved a test score mean of 93.73%, and SVM had a test score mean of 96.44%. The superior performance of LR can be attributed to its simplicity and interpretability, which provide a clear understanding of the relationship between input features and the outcome. This is particularly valuable in the medical domain, where interpretability is crucial for decision-making. Moreover, the computational efficiency of LR offers advantages in terms of scalability and real-world applicability. The results of this study highlight the power of simplicity in the context of breast cancer diagnosis and suggest that simpler linear models like LR can be more effective, interpretable, and computationally efficient than their complex counterparts, making them a more suitable choice for medical applications.

Motivation & Objective

• To evaluate the performance of simple linear models versus complex machine learning techniques in breast cancer diagnosis.
• To assess the trade-offs between model accuracy, interpretability, and computational efficiency in medical AI applications.
• To determine whether simpler models like Logistic Regression can outperform more complex models such as Decision Trees and SVMs in clinical diagnostic tasks.
• To emphasize the importance of interpretability and efficiency in real-world medical decision-making systems.
• To advocate for the use of simpler, more transparent models in healthcare settings where model trustworthiness and scalability are critical.

Proposed method

• Used the UCI Breast Cancer Wisconsin (Diagnostic) dataset containing 569 instances with 32 attributes, including diagnosis (malignant/benign) and 9 real-valued features derived from fine needle aspirate images.
• Trained and evaluated three models: Logistic Regression (LR), Decision Trees (DT), and Support Vector Machines (SVM) using standard machine learning pipelines.
• Optimized model performance through hyperparameter tuning and used 5-fold cross-validation to ensure robust evaluation metrics.
• Measured performance using test accuracy (mean and standard deviation), computation time, and generalization capability (train-test score gap).
• Assessed interpretability by analyzing feature coefficients in LR and decision rules in DT, while evaluating SVM’s performance with kernel functions for non-linear boundaries.
• Compared model efficiency via inference time (35.56 ms for LR) and evaluated overfitting risks using the difference between training and test scores.

Figure 1: Parallel coordinate plot of the Breast Cancer Wisconsin (Diagnostic) Data Set.

Experimental results

Research questions

• RQ1 Does a simple linear model like Logistic Regression outperform more complex models such as Decision Trees and SVMs in breast cancer diagnosis?
• RQ2 How do interpretability and computational efficiency compare between simple linear models and complex machine learning techniques in medical applications?
• RQ3 To what extent do complex models like SVM and DT suffer from overfitting or increased inference time compared to simpler models?
• RQ4 Can simpler models maintain high accuracy while offering greater transparency and easier integration into clinical workflows?
• RQ5 What is the trade-off between model complexity, performance, and practicality in real-world medical diagnosis systems?

Key findings

• Logistic Regression (LR) achieved the highest test accuracy at 97.28% with a standard deviation of 1.62%, outperforming both Decision Trees (93.73%) and SVM (96.44%).
• LR demonstrated superior computational efficiency with an inference time of only 35.56 ms, significantly faster than SVM and competitive with DT.
• Decision Trees showed faster computation than LR but exhibited signs of overfitting, indicated by a larger gap between training and test scores.
• SVM achieved high accuracy (96.44%) but required significantly longer computation time, limiting its scalability in time-sensitive clinical applications.
• The interpretability of LR—through clear feature coefficients—provided a transparent decision-making process, which is critical in medical diagnostics.
• Despite their complexity, DT and SVM did not achieve better performance than LR, and their drawbacks in overfitting (DT) and speed (SVM) reduced their practical utility in this context.

Figure 2: Distribution of each feature for both benign and malignant instances.