[Paper Review] A Comparative Evaluation of Quantification Methods

Tobias Schumacher, Markus Strohmaier+1 more|arXiv (Cornell University)|Mar 4, 2021

Imbalanced Data Classification Techniques28 references4 citations

TL;DR

This paper presents a comprehensive empirical evaluation of 24 quantification methods across 40 datasets, comparing binary and multiclass settings under varying distribution shifts and training set sizes. It identifies Median Sweep, TSMax, DyS, and Friedman’s method as top performers in binary quantification, while Generalized Probabilistic Adjusted Count, readme, energy distance minimization, EM for quantification, and Friedman’s method excel in multiclass settings, with multiclass performance significantly worse than binary due to inherent complexity.

ABSTRACT

Quantification represents the problem of estimating the distribution of class labels on unseen data. It also represents a growing research field in supervised machine learning, for which a large variety of different algorithms has been proposed in recent years. However, a comprehensive empirical comparison of quantification methods that supports algorithm selection is not available yet. In this work, we close this research gap by conducting a thorough empirical performance comparison of 24 different quantification methods on overall more than 40 data sets, considering binary as well as multiclass quantification settings. We observe that no single algorithm generally outperforms all competitors, but identify a group of methods including the threshold selection-based Median Sweep and TSMax methods, the DyS framework including the HDy method, Forman's mixture model, and Friedman's method that performs best in the binary setting. For the multiclass setting, we observe that a different, broad group of algorithms yields good performance, including the HDx method, the Generalized Probabilistic Adjusted Count, the readme method, the energy distance minimization method, the EM algorithm for quantification, and Friedman's method. We also find that tuning the underlying classifiers has in most cases only a limited impact on the quantification performance. More generally, we find that the performance on multiclass quantification is inferior to the results obtained in the binary setting. Our results can guide practitioners who intend to apply quantification algorithms and help researchers to identify opportunities for future research.

Motivation & Objective

• To address the lack of comprehensive, empirical comparisons of quantification methods in machine learning.
• To evaluate the performance of 24 state-of-the-art quantification algorithms across diverse datasets and settings, including binary and multiclass quantification.
• To investigate the impact of distribution shifts between training and test data, training set size, and classifier hyperparameter tuning on quantification performance.
• To identify robust, general-purpose quantification methods suitable for practical deployment.
• To highlight research gaps, particularly the underperformance of multiclass quantification and the limited benefit of classifier tuning.

Proposed method

• The study evaluates 24 quantification methods, including threshold-based methods (Median Sweep, TSMax), distribution matching (DyS, energy distance), probabilistic adjustment (Generalized Probabilistic Adjusted Count), and iterative algorithms (EM for quantification, readme).
• Experiments are conducted on 40 datasets, covering both binary and multiclass settings, with controlled distribution shifts between training and test data.
• Performance is measured across varying training set sizes and multiple sampling iterations (10 per experiment), totaling over 5 million algorithm runs.
• Classifier hyperparameter tuning is evaluated separately for a subset of datasets due to computational constraints, assessing its impact on quantification accuracy.
• A case study is conducted on the LeQua challenge dataset to validate findings in a real-world scenario with extensive validation data.
• Statistical significance testing is used to compare method performance, with average rankings and effect sizes reported.

Experimental results

Research questions

• RQ1 Which quantification methods consistently outperform others in binary and multiclass settings under varying distribution shifts and training set sizes?
• RQ2 How does tuning the underlying base classifiers affect the performance of quantification algorithms?
• RQ3 To what extent does the multiclass quantification problem remain more challenging than the binary case, and why?
• RQ4 Do methods based on the classify-and-count principle, even when optimized, perform competitively compared to specialized quantification frameworks?
• RQ5 How generalizable are the findings from the main experiments to real-world applications, as validated by a case study on the LeQua challenge dataset?

Key findings

• In the binary setting, Median Sweep, TSMax, DyS, and Friedman’s method consistently rank among the top performers, with statistically significant superiority over other methods.
• In the multiclass setting, Generalized Probabilistic Adjusted Count, readme, energy distance minimization, EM for quantification, and Friedman’s method show significantly better average performance than other approaches.
• Multiclass quantification performance is substantially worse than binary quantification, with error scores multiple times higher, indicating a harder problem requiring more research attention.
• Tuning the hyperparameters of underlying base classifiers has only a limited impact on final quantification performance, both in binary and multiclass settings.
• Methods based on the classify-and-count principle, even when the base classifier is optimized for quantification, perform worse on average than specialized frameworks, suggesting inherent limitations in this approach.
• The case study on the LeQua challenge dataset confirms the general trends observed in the main experiments, validating the robustness of the findings across diverse real-world scenarios.