[Paper Review] A Comprehensive Survey on Heart Sound Analysis in the Deep Learning Era

Zhao Ren, Yi Chang+4 more|arXiv (Cornell University)|Jan 23, 2023

Phonocardiography and Auscultation Techniques4 citations

TL;DR

This survey provides a comprehensive overview of deep learning applications in heart sound analysis from 2017 to 2022, comparing classic machine learning with deep learning approaches. It covers denoising, segmentation, classification, and interpretability, highlighting state-of-the-art models, publicly available datasets, code repositories, and future research directions including foundation models and human-AI collaboration.

ABSTRACT

Heart sound auscultation has been applied in clinical usage for early screening of cardiovascular diseases. Due to the high demand for auscultation expertise, automatic auscultation can help with auxiliary diagnosis and reduce the burden of training professional clinicians. Nevertheless, there is a limit to classic machine learning's performance improvement in the era of big data. Deep learning has outperformed classic machine learning in many research fields, as it employs more complex model architectures with a stronger capability of extracting effective representations. Moreover, it has been successfully applied to heart sound analysis in the past years. As most review works about heart sound analysis were carried out before 2017, the present survey is the first to work on a comprehensive overview to summarise papers on heart sound analysis with deep learning published in 2017--2022. This work introduces both classic machine learning and deep learning for comparison, and further offer insights about the advances and future research directions in deep learning for heart sound analysis. Our repository is publicly available at \url{https://github.com/zhaoren91/awesome-heart-sound-analysis}.

Motivation & Objective

• To provide a systematic and up-to-date review of deep learning techniques in heart sound analysis, focusing on the period 2017–2022.
• To compare the performance and limitations of classic machine learning versus deep learning in heart sound processing tasks.
• To summarize state-of-the-art methods in heart sound denoising, segmentation, classification, and model interpretability.
• To identify key challenges and future research directions, including foundation models, human-machine collaboration, and model dependability.
• To compile publicly available datasets, evaluation metrics, and open-source code for reproducibility and community advancement.

Proposed method

• The survey conducts a systematic literature review of peer-reviewed papers on heart sound analysis using deep learning, focusing on works published between 2017 and 2022.
• It compares classic machine learning methods (e.g., SVMs, HMMs) with deep learning models that use raw audio or spectrograms as input, bypassing hand-crafted feature engineering.
• The study evaluates deep learning architectures such as CNNs, RNNs, Transformers, and self-supervised models like Wav2Vec and HuBERT for heart sound tasks.
• It analyzes interpretability techniques including attention visualization, SHAP, and prototype-based explanations (e.g., prototypes, criticisms, counterfactuals) to enhance model transparency.
• The survey discusses emerging trends such as federated learning for data privacy, machine unlearning, and adversarial robustness to improve model dependability.
• It evaluates publicly available datasets (e.g., PhysioNet, LIDC, CRESCENDO) and code repositories to support reproducibility and benchmarking.

Experimental results

Research questions

• RQ1 How have deep learning models outperformed classic machine learning in heart sound denoising, segmentation, and classification tasks between 2017 and 2022?
• RQ2 What are the most effective deep learning architectures and input representations (e.g., raw waveforms, spectrograms) for heart sound analysis?
• RQ3 How can model interpretability be enhanced to improve trust and clinical adoption in automated heart sound diagnosis?
• RQ4 What role can foundation models pre-trained on large-scale audio data play in advancing heart sound analysis?
• RQ5 What are the key challenges in deploying reliable, privacy-preserving, and robust deep learning systems for real-world clinical use?

Key findings

• Deep learning models significantly outperform classic machine learning in heart sound classification, especially on large-scale datasets, due to their ability to learn hierarchical representations without manual feature engineering.
• Self-supervised and pre-trained models such as Wav2Vec and HuBERT have shown strong performance in few-shot and zero-shot learning scenarios, reducing reliance on large annotated datasets.
• Attention mechanisms and SHAP-based explanations enable local interpretability by highlighting relevant segments of heart sounds contributing to model predictions.
• Prototype-based methods (e.g., prototypes, criticisms, counterfactuals) offer a promising path toward global interpretability by revealing typical patterns of each heart sound class.
• Federated learning and machine unlearning are emerging as critical techniques for preserving patient data privacy while maintaining diagnostic accuracy.
• Adversarial robustness and model dependability remain open challenges, with increasing need for secure and trustworthy AI in clinical decision support systems.