[Paper Review] Deciding when to stop: Efficient stopping of active learning guided drug-target prediction

Maja Temerinac-Ott, Armaghan W. Naik+1 more|arXiv (Cornell University)|Apr 9, 2015

Computational Drug Discovery Methods29 references3 citations

TL;DR

This paper proposes a simulation-based regression model to predict active learning accuracy in drug-target interaction prediction, enabling early stopping with minimal performance loss. By training on simulated data, the method reduces experimental costs by up to 40% while maintaining high predictive accuracy on real-world datasets.

ABSTRACT

Active learning has shown to reduce the number of experiments needed to obtain high-confidence drug-target predictions. However, in order to actually save experiments using active learning, it is crucial to have a method to evaluate the quality of the current prediction and decide when to stop the experimentation process. Only by applying reliable stoping criteria to active learning, time and costs in the experimental process can be actually saved. We compute active learning traces on simulated drug-target matrices in order to learn a regression model for the accuracy of the active learner. By analyzing the performance of the regression model on simulated data, we design stopping criteria for previously unseen experimental matrices. We demonstrate on four previously characterized drug effect data sets that applying the stopping criteria can result in upto 40% savings of the total experiments for highly accurate predictions.

Motivation & Objective

• To address the critical challenge of determining when to stop active learning in drug-target prediction to avoid unnecessary experiments.
• To develop a reliable, data-driven stopping criterion that minimizes experimental cost without sacrificing predictive accuracy.
• To calibrate stopping rules using simulated active learning traces to generalize to real-world drug-target interaction matrices.
• To demonstrate that early stopping based on predicted model accuracy can achieve comparable performance with significantly fewer experiments.
• To provide a practical framework applicable beyond kernelized matrix factorization for active learning in high-throughput biological screening.

Proposed method

• Simulate drug-target interaction matrices using known biological data to generate active learning traces with iterative model updates.
• Extract trajectory features (e.g., uncertainty, label consistency, prediction variance) from simulated active learning processes.
• Train a regression model on simulated data to predict the true accuracy of the active learner on unseen test sets.
• Apply the trained regression model to real-world drug-target datasets to estimate current model performance during active learning.
• Use the predicted accuracy to trigger stopping when further experiments are unlikely to improve performance significantly.
• Compare multiple stopping criteria (e.g., uncertainty, expected error, predicted accuracy) using a benchmark based on the best stopping time (BST).

Experimental results

Research questions

• RQ1 Can a regression model trained on simulated active learning traces accurately predict the true accuracy of a drug-target prediction model during active learning?
• RQ2 Which stopping criterion—based on uncertainty, expected error, or predicted accuracy—most closely approximates the optimal stopping time (BST) in real-world drug-target prediction?
• RQ3 To what extent can experimental costs be reduced by applying a simulation-derived stopping rule without degrading prediction performance?
• RQ4 How does the performance of the proposed stopping rule compare to fixed-threshold and adaptive-threshold methods in terms of proximity to the best stopping time?
• RQ5 Can the framework be generalized beyond kernelized matrix factorization to other drug-target prediction models?

Key findings

• The predicted accuracy (PA) stopping rule with a threshold of 0.9 achieved the smallest average difference (13.7%) from the best stopping time (BST) across four real-world datasets.
• The PA method reduced the number of required experiments by up to 40% while maintaining high predictive accuracy, with AUC values within 1–3% of the full experiment baseline.
• The OU (overall uncertainty) and MEE (minimum expected error) methods with fixed thresholds performed significantly worse, with average differences from BST exceeding 30%.
• Adaptive thresholding improved OU and MEE performance but still underperformed the PA method, which showed the most consistent and accurate stopping behavior.
• The proposed method achieved 99% accuracy on three high-uniqueness datasets using only 2–3 times fewer experiments than random sampling, demonstrating strong efficiency gains.
• The regression model trained on simulated data provided a reliable lower-bound estimate of true model accuracy, enabling effective early stopping in real-world applications.