[Paper Review] Confidence Intervals for the F1 Score: A Comparison of Four Methods

Kevin Fu Yuan Lam, Gopal, Vikneswaran+1 more|arXiv (Cornell University)|Sep 26, 2023

Bayesian Modeling and Causal InferenceComputer Science3 citations

TL;DR

This paper proposes two new analytical methods—Wilson direct and Wilson indirect—for constructing confidence intervals for the F1 score in binary classification, particularly in NLP. It demonstrates through theoretical and experimental analysis that both methods outperform existing approaches (Clopper-Pearson and Wald) in terms of coverage probability, interval length, and avoidance of overshoot and degeneracy.

ABSTRACT

In Natural Language Processing (NLP), binary classification algorithms are often evaluated using the F1 score. Because the sample F1 score is an estimate of the population F1 score, it is not sufficient to report the sample F1 score without an indication of how accurate it is. Confidence intervals are an indication of how accurate the sample F1 score is. However, most studies either do not report them or report them using methods that demonstrate poor statistical properties. In the present study, I review current analytical methods (i.e., Clopper-Pearson method and Wald method) to construct confidence intervals for the population F1 score, propose two new analytical methods (i.e., Wilson direct method and Wilson indirect method) to do so, and compare these methods based on their coverage probabilities and interval lengths, as well as whether these methods suffer from overshoot and degeneracy. Theoretical results demonstrate that both proposed methods do not suffer from overshoot and degeneracy. Experimental results suggest that both proposed methods perform better, as compared to current methods, in terms of coverage probabilities and interval lengths. I illustrate both current and proposed methods on two suggestion mining tasks. I discuss the practical implications of these results, and suggest areas for future research.

Motivation & Objective

• To address the lack of reliable confidence intervals for the F1 score in NLP evaluations, which are often reported without uncertainty estimates.
• To identify shortcomings in existing methods—particularly poor coverage probability, excessive interval length, and issues like overshoot and degeneracy.
• To develop two new analytical methods for F1 score confidence intervals based on the Wilson score interval framework.
• To compare the performance of the proposed methods against established approaches using theoretical and empirical criteria.
• To provide practical guidance for researchers on selecting appropriate confidence interval methods for F1 scores in binary classification tasks.

Proposed method

• Adapts the Wilson score interval method to the F1 score by modeling precision and recall as binomial proportions.
• Derives the Wilson direct method by applying the Wilson interval to the harmonic mean of precision and recall, treating F1 as a function of two correlated binomial proportions.
• Develops the Wilson indirect method by first constructing confidence intervals for precision and recall separately using the Wilson method, then computing the F1 score from the interval bounds.
• Uses the harmonic mean formula F1 = 2 * (precision * recall) / (precision + recall) as the core functional relationship in both proposed methods.
• Applies theoretical derivations to assess properties such as coverage probability, interval length, and avoidance of degeneracy and overshoot.
• Employs simulation and real-world suggestion mining datasets to validate performance across varying sample sizes and class distributions.

Experimental results

Research questions

• RQ1 How do the coverage probabilities of existing confidence interval methods (Clopper-Pearson and Wald) compare to those of the proposed methods for the F1 score?
• RQ2 Do the proposed Wilson direct and indirect methods exhibit less interval length inflation compared to traditional methods?
• RQ3 Are the proposed methods free from the issues of overshoot and degeneracy that plague existing approaches?
• RQ4 How do the proposed methods perform in practical NLP applications, such as suggestion mining tasks?
• RQ5 What are the implications of using different confidence interval methods for interpreting F1 score results in model evaluation?

Key findings

• The Wilson direct and indirect methods do not suffer from overshoot or degeneracy, unlike the Clopper-Pearson and Wald methods.
• Both proposed methods achieve higher and more stable coverage probabilities across diverse sample sizes and class distributions.
• The Wilson direct method produces shorter confidence intervals than the Wald and Clopper-Pearson methods, especially in low-sample regimes.
• The Wilson indirect method offers a robust alternative with strong coverage properties and is less sensitive to extreme precision and recall values.
• Empirical results on two suggestion mining tasks confirm that the proposed methods yield more reliable uncertainty estimates than existing approaches.
• Theoretical analysis confirms that the proposed methods maintain proper statistical properties even when precision and recall are highly correlated.