[Paper Review] Unsupervised Evaluation and Weighted Aggregation of Ranked Predictions

Mehmet Eren Ahsen, Robert Vogel+1 more|arXiv (Cornell University)|Feb 13, 2018

Machine Learning and Data Classification27 references4 citations

TL;DR

This paper proposes SUMMA, a novel unsupervised framework for evaluating and aggregating ranked predictions from multiple base classifiers without labeled data. By estimating classifier performance and learning optimal ensemble weights using only unlabeled data, SUMMA achieves superior predictive accuracy in binary classification tasks, demonstrating strong performance even when labeled data is unavailable.

ABSTRACT

Learning algorithms that aggregate predictions from an ensemble of diverse base classifiers consistently outperform individual methods. Many of these strategies have been developed in a supervised setting, where the accuracy of each base classifier can be empirically measured and this information is incorporated in the training process. However, the reliance on labeled data precludes the application of ensemble methods to many real world problems where labeled data has not been curated. To this end we developed a new theoretical framework for binary classification, the Strategy for Unsupervised Multiple Method Aggregation (SUMMA), to estimate the performances of base classifiers and an optimal strategy for ensemble learning from unlabeled data.

Motivation & Objective

• To address the challenge of ensemble learning in scenarios where labeled data is unavailable or impractical to obtain.
• To develop a theoretical framework that enables unsupervised estimation of base classifier performance.
• To design a weighted aggregation strategy that optimally combines diverse classifiers using only unlabeled data.
• To enable reliable model stacking and ensemble prediction in real-world applications with no ground-truth labels.
• To provide a principled alternative to supervised ensemble methods in low-resource or unlabeled data environments.

Proposed method

• Proposes the Strategy for Unsupervised Multiple Method Aggregation (SUMMA), a theoretical framework for unsupervised evaluation of binary classifiers.
• Uses the rank-ordering of predictions across multiple classifiers to infer relative performance without labels.
• Employs a statistical model to estimate the reliability and accuracy of each base classifier based on prediction consistency and ranking patterns.
• Derives optimal ensemble weights by minimizing an unsupervised loss function that reflects expected generalization error.
• Applies a weighted aggregation scheme where classifier weights are learned from unlabeled data using rank-based performance estimation.
• Leverages the structure of prediction scores to infer calibration and discrimination ability in the absence of labels.

Experimental results

Research questions

• RQ1 Can classifier performance be reliably estimated in the absence of labeled data?
• RQ2 How can optimal ensemble weights be learned without ground-truth labels?
• RQ3 What is the theoretical foundation for unsupervised evaluation of ranked predictions in binary classification?
• RQ4 Can unsupervised aggregation outperform supervised baselines when labels are unavailable?
• RQ5 How does prediction ranking across multiple models inform performance estimation and ensemble weighting?

Key findings

• SUMMA successfully estimates classifier performance with high correlation to true accuracy when labels are available, validating its unsupervised estimation capability.
• The method achieves competitive or superior predictive performance compared to supervised ensemble baselines in low-label regimes.
• The framework demonstrates robustness across diverse datasets and classifier types, even with significant model heterogeneity.
• The unsupervised weighting scheme significantly improves ensemble accuracy over uniform or heuristic weighting strategies.
• Prediction ranking patterns across models provide sufficient information to infer relative classifier quality without labels.
• The approach enables effective model stacking in real-world applications where labeling is costly or infeasible.