[Paper Review] Multilabel Consensus Classification

Sihong Xie, Xiangnan Kong+3 more|arXiv (Cornell University)|Oct 16, 2013

Text and Document Classification Technologies13 references3 citations

TL;DR

This paper proposes MLCM-r and MLCM-a, two novel multilabel consensus classification algorithms that jointly model label correlations and model consensus to optimize ranking loss and microAUC, respectively. The methods outperform baselines with up to 45% lower ranking loss and 20% higher microAUC on multilabel tasks without requiring access to raw training or test data.

ABSTRACT

In the era of big data, a large amount of noisy and incomplete data can be collected from multiple sources for prediction tasks. Combining multiple models or data sources helps to counteract the effects of low data quality and the bias of any single model or data source, and thus can improve the robustness and the performance of predictive models. Out of privacy, storage and bandwidth considerations, in certain circumstances one has to combine the predictions from multiple models or data sources to obtain the final predictions without accessing the raw data. Consensus-based prediction combination algorithms are effective for such situations. However, current research on prediction combination focuses on the single label setting, where an instance can have one and only one label. Nonetheless, data nowadays are usually multilabeled, such that more than one label have to be predicted at the same time. Direct applications of existing prediction combination methods to multilabel settings can lead to degenerated performance. In this paper, we address the challenges of combining predictions from multiple multilabel classifiers and propose two novel algorithms, MLCM-r (MultiLabel Consensus Maximization for ranking) and MLCM-a (MLCM for microAUC). These algorithms can capture label correlations that are common in multilabel classifications, and optimize corresponding performance metrics. Experimental results on popular multilabel classification tasks verify the theoretical analysis and effectiveness of the proposed methods.

Motivation & Objective

• Address the challenge of combining multilabel predictions from multiple models when training and test data are inaccessible due to privacy, bandwidth, or storage constraints.
• Overcome the limitation of existing prediction combination methods, which are designed for single-label classification and degrade in performance on multilabel settings.
• Develop algorithms that explicitly model label correlations using only the predictions of base models to improve multilabel classification performance.
• Optimize for specific, widely used multilabel evaluation metrics—ranking loss and microAUC—by aligning the consensus mechanism with the underlying label correlation structure.
• Provide theoretical justification that the proposed methods are optimal for their respective metrics, filling a gap in multilabel consensus learning.

Proposed method

• MLCM-r uses random walk in the label space to infer label correlations and consolidate predictions, with the objective of minimizing ranking loss.
• The algorithm models label correlation as a Markov chain over the label graph, where transition probabilities are derived from base model predictions.
• MLCM-a formulates prediction consolidation as an optimization problem that regularizes predictions using partial correlations between labels to maximize microAUC.
• The objective function in MLCM-a is designed such that its solution asymptotically optimizes microAUC by capturing conditional dependencies among labels.
• Both algorithms operate solely on the predictions of base models, avoiding the need for raw data, and are scalable to large multilabel datasets.
• Theoretical analysis proves that MLCM-r minimizes ranking loss and MLCM-a maximizes microAUC under the proposed formulations.

Experimental results

Research questions

• RQ1 Can prediction combination methods be effectively extended to multilabel classification when only model predictions are available and raw data is inaccessible?
• RQ2 How can label correlations be modeled and exploited in multilabel consensus classification without access to training or test data?
• RQ3 Can consensus-based algorithms be formally linked to specific multilabel evaluation metrics such as ranking loss and microAUC?
• RQ4 Is it possible to design optimization frameworks that jointly model label correlations and model consensus to improve multilabel prediction performance?
• RQ5 Do the proposed algorithms outperform existing baselines in terms of both ranking loss and microAUC on real-world multilabel datasets?

Key findings

• MLCM-r reduces ranking loss by up to 45% compared to baseline methods on six multilabel classification tasks.
• MLCM-a increases microAUC by up to 20% compared to baselines, demonstrating superior performance on the microAUC metric.
• Theoretical analysis confirms that MLCM-r minimizes ranking loss and MLCM-a maximizes microAUC under the proposed optimization frameworks.
• Experiments show that modeling label correlations significantly improves performance over methods that treat labels independently.
• The proposed methods outperform three baselines across all six multilabel datasets tested, validating their robustness and generalization.
• The performance gains are consistent across diverse multilabel tasks, including text and image categorization, bioinformatics, and document tagging.