[Paper Review] When and Why does a Model Fail? A Human-in-the-loop Error Detection Framework for Sentiment Analysis

Zhe Liu, Yufan Guo+1 more|arXiv (Cornell University)|Jun 2, 2021

Adversarial Robustness in Machine Learning21 references5 citations

TL;DR

This paper proposes a human-in-the-loop error detection framework for sentiment analysis that identifies erroneous model predictions using explainable features. By combining global feature relevance assessment with local feature contribution analysis, the framework achieves high-precision error detection on unseen data with minimal human effort, outperforming uncertainty sampling by 11 percentage points in precision at top-100 predictions.

ABSTRACT

Although deep neural networks have been widely employed and proven effective in sentiment analysis tasks, it remains challenging for model developers to assess their models for erroneous predictions that might exist prior to deployment. Once deployed, emergent errors can be hard to identify in prediction run-time and impossible to trace back to their sources. To address such gaps, in this paper we propose an error detection framework for sentiment analysis based on explainable features. We perform global-level feature validation with human-in-the-loop assessment, followed by an integration of global and local-level feature contribution analysis. Experimental results show that, given limited human-in-the-loop intervention, our method is able to identify erroneous model predictions on unseen data with high precision.

Motivation & Objective

• To address the challenge of detecting erroneous predictions in deployed sentiment models before they cause harm.
• To reduce reliance on costly, instance-level human labeling by enabling human assessment at the global feature level.
• To improve model accountability by providing interpretable explanations for why predictions fail.
• To proactively detect errors—even in high-confidence predictions—where traditional uncertainty sampling fails.
• To enable scalable, efficient error detection that generalizes to unseen data without ground truth.

Proposed method

• The framework first computes local feature contributions per instance using data perturbation to measure feature impact on predictions.
• It aggregates local contributions into global feature importance scores across the test set to identify consistently influential features.
• Human annotators evaluate the relevance of top global features to sentiment classes, flagging those deemed irrelevant or misleading.
• An erroneous score is computed by combining global feature relevance with local prediction confidence, using a threshold-based flagging mechanism.
• The method leverages human-in-the-loop validation at the feature level rather than instance level, reducing annotation burden.
• The framework integrates both global and local analyses to detect errors even when model confidence is high.

Experimental results

Research questions

• RQ1 How can we detect erroneous predictions in sentiment models with minimal human intervention?
• RQ2 Can global feature-level human assessment outperform instance-level labeling in error detection efficiency and accuracy?
• RQ3 Can the framework detect errors that uncertainty sampling misses, especially in high-confidence predictions?
• RQ4 How effective is the integration of global feature relevance and local prediction contributions in identifying model failures?
• RQ5 Can the framework be used to detect model bias through problematic global features?

Key findings

• The proposed method achieved 82.0% precision@K at K=100, significantly outperforming uncertainty sampling’s 71.0% precision@K.
• At K=400, the precision gap between the two methods narrowed to only 0.6%, indicating diminishing returns as thresholds were relaxed.
• Approximately 40% of the erroneous instances detected by the framework had prediction probabilities above 0.7, showing it detects errors even when models are highly confident.
• Human assessors, including non-experts, achieved high inter-rater agreement in assessing global feature relevance, validating the feasibility of feature-level labeling.
• The framework successfully identified problematic global features such as 'netflix', 'dems', and 'palestine' being incorrectly associated with negative sentiment.
• The method enables proactive detection of errors on unseen data without requiring ground truth, supporting continuous model refinement.