[Paper Review] SoCal: Selective Oracle Questioning for Consistency-based Active Learning of Cardiac Signals

Dani Kiyasseh, Tingting Zhu+1 more|arXiv (Cornell University)|May 4, 2021

Machine Learning and Algorithms47 references4 citations

TL;DR

SoCal is a consistency-based active learning framework for cardiac signal analysis that reduces physician labeling burden by dynamically choosing between querying an oracle or generating pseudo-labels. It maintains strong model performance even with a noisy oracle, significantly decreasing reliance on human annotations while preserving accuracy.

ABSTRACT

The ubiquity and rate of collection of cardiac signals produce large, unlabelled datasets. Active learning (AL) can exploit such datasets by incorporating human annotators (oracles) to improve generalization performance. However, the over-reliance of existing algorithms on oracles continues to burden physicians. To minimize this burden, we propose SoCal, a consistency-based AL framework that dynamically determines whether to request a label from an oracle or to generate a pseudo-label instead. We show that our framework decreases the labelling burden while maintaining strong performance, even in the presence of a noisy oracle.

Motivation & Objective

• To reduce the labeling burden on physicians in cardiac signal analysis by minimizing reliance on human oracles.
• To maintain high model performance despite noisy or unreliable oracle feedback.
• To develop a dynamic decision mechanism that selects between oracle queries and pseudo-labeling based on prediction consistency.
• To enable efficient active learning in large-scale, unlabelled cardiac signal datasets.
• To improve generalization in cardiac signal classification with minimal human intervention.

Proposed method

• SoCal uses a consistency-based criterion to evaluate model predictions across multiple forward passes with data augmentations.
• If predictions are consistent across augmented views, the model generates a pseudo-label instead of querying the oracle.
• When predictions are inconsistent, the system queries the oracle for a ground-truth label.
• The framework integrates uncertainty and consistency metrics to guide the decision between querying and pseudo-labeling.
• It employs a dynamic thresholding mechanism to balance exploration (querying) and exploitation (pseudo-labeling).
• The method is trained end-to-end with a loss function that combines supervised and pseudo-supervised learning components.

Experimental results

Research questions

• RQ1 How can active learning in cardiac signal classification reduce physician labeling burden without sacrificing model performance?
• RQ2 To what extent can pseudo-labeling replace oracle queries in consistency-based active learning for cardiac signals?
• RQ3 How does the framework perform under noisy oracle conditions compared to traditional active learning?
• RQ4 What is the optimal balance between querying and pseudo-labeling for improving model generalization?
• RQ5 Can dynamic decision-making based on prediction consistency improve efficiency in cardiac signal annotation?

Key findings

• SoCal significantly reduces the number of oracle queries required during training, decreasing labeling burden.
• The framework maintains high classification accuracy even when the oracle provides noisy labels.
• Dynamic pseudo-labeling based on prediction consistency leads to improved model generalization over time.
• The method outperforms standard active learning baselines in terms of label efficiency and performance stability.
• Consistency-based decision-making enables reliable model updates with minimal human intervention.
• The approach is robust to label noise, making it suitable for real-world clinical deployment.