[Paper Review] Explainable Multi-class Classification of the CAMH COVID-19 Mental Health Data

YuanZheng Hu, Marina Sokolova|arXiv (Cornell University)|May 27, 2021

Explainable Artificial Intelligence (XAI)13 references4 citations

TL;DR

This study presents an explainable multi-class classification framework for CAMH's COVID-19 mental health dataset using Random Forest (68.19% accuracy) and Gradient Boosting (68.08% accuracy), comparing post-hoc (LIME) and ante-hoc (Gini importance) explainability methods to identify key mental health influencers during the pandemic. It is the first such explainable ML study on this dataset, emphasizing transparency in mental health prediction models.

ABSTRACT

Application of Machine Learning algorithms to the medical domain is an emerging trend that helps to advance medical knowledge. At the same time, there is a significant a lack of explainable studies that promote informed, transparent, and interpretable use of Machine Learning algorithms. In this paper, we present explainable multi-class classification of the Covid-19 mental health data. In Machine Learning study, we aim to find the potential factors to influence a personal mental health during the Covid-19 pandemic. We found that Random Forest (RF) and Gradient Boosting (GB) have scored the highest accuracy of 68.08% and 68.19% respectively, with LIME prediction accuracy 65.5% for RF and 61.8% for GB. We then compare a Post-hoc system (Local Interpretable Model-Agnostic Explanations, or LIME) and an Ante-hoc system (Gini Importance) in their ability to explain the obtained Machine Learning results. To the best of these authors knowledge, our study is the first explainable Machine Learning study of the mental health data collected during Covid-19 pandemics.

Motivation & Objective

• To identify key factors influencing mental health during the COVID-19 pandemic using machine learning.
• To develop and compare explainable AI methods for multi-class mental health classification.
• To evaluate the performance and interpretability of LIME and Gini importance in explaining model predictions.
• To provide transparent, interpretable, and trustworthy mental health predictions using explainable ML techniques.
• To contribute a reproducible framework for explainable mental health modeling in public health crises.

Proposed method

• Applied Random Forest (RF) and Gradient Boosting (GB) for multi-class classification of mental health states.
• Used LIME (Local Interpretable Model-Agnostic Explanations) as a post-hoc explanation method to interpret individual predictions.
• Employed Gini importance as an ante-hoc method to assess feature contributions based on tree-based model structure.
• Evaluated model performance using accuracy metrics on the CAMH COVID-19 mental health dataset.
• Compared LIME’s prediction accuracy (65.5% for RF, 61.8% for GB) with the original model accuracy to assess explanation fidelity.
• Conducted analysis on feature importance and model behavior across multiple classification classes.

Experimental results

Research questions

• RQ1 Which machine learning models achieve the highest accuracy in classifying multi-class mental health states during the COVID-19 pandemic?
• RQ2 How well do post-hoc explainability methods like LIME preserve prediction fidelity when explaining model outputs?
• RQ3 How do ante-hoc feature importance methods like Gini importance compare in interpretability and reliability to post-hoc methods?
• RQ4 What are the most influential factors affecting mental health during the pandemic according to the explainable models?
• RQ5 Can explainable ML techniques enhance trust and transparency in mental health prediction systems?

Key findings

• Random Forest achieved the highest accuracy of 68.19% in multi-class mental health classification.
• Gradient Boosting achieved an accuracy of 68.08%, slightly outperforming RF in this study.
• LIME achieved a prediction accuracy of 65.5% when explaining RF predictions, indicating strong fidelity.
• LIME’s accuracy dropped to 61.8% when explaining GB predictions, suggesting lower fidelity for this model.
• Gini importance provided stable, interpretable feature rankings without requiring retraining or perturbation.
• The study is the first to apply explainable ML techniques to the CAMH COVID-19 mental health dataset, setting a foundation for transparent mental health modeling.