[Paper Review] Topic Model Based Multi-Label Classification from the Crowd

Divya Padmanabhan, Satyanath Bhat+2 more|arXiv (Cornell University)|Apr 4, 2016

Text and Document Classification Technologies1 references3 citations

TL;DR

This paper proposes ML-PA-LDA, a novel topic model for multi-label classification that explicitly models both the presence and absence of labels to capture label correlations. It further extends this to ML-PA-LDA-C, which handles noisy, heterogeneous crowdworker labels and jointly learns annotator quality, achieving superior performance on real-world datasets even with minimal training data.

ABSTRACT

Multi-label classification is a common supervised machine learning problem where each instance is associated with multiple classes. The key challenge in this problem is learning the correlations between the classes. An additional challenge arises when the labels of the training instances are provided by noisy, heterogeneous crowdworkers with unknown qualities. We first assume labels from a perfect source and propose a novel topic model where the present as well as the absent classes generate the latent topics and hence the words. We non-trivially extend our topic model to the scenario where the labels are provided by noisy crowdworkers. Extensive experimentation on real world datasets reveals the superior performance of the proposed model. The proposed model learns the qualities of the annotators as well, even with minimal training data.

Motivation & Objective

• To address the challenge of modeling label correlations in multi-label classification, particularly the informative role of absent labels.
• To develop a generative topic model that incorporates both presence and absence of classes to improve classification performance.
• To extend the model to handle noisy, heterogeneous crowd labels where annotator quality is unknown and must be learned.
• To enable effective learning from partial labeling, where annotators may not label all classes for a document.
• To validate the model’s robustness and accuracy on real-world datasets with varying training data sizes and annotator qualities.

Proposed method

• Proposes ML-PA-LDA, a topic model where both present and absent labels contribute to topic generation, modeling label correlations more effectively than prior methods.
• Introduces a non-conjugate variational EM algorithm to infer latent topics, label assignments, and annotator qualities in the presence of noisy crowd labels.
• Models each document as a mixture of topics, where each topic is associated with a multinomial distribution over words and a multinomial distribution over label presence/absence.
• Incorporates annotator-specific error models by learning individual annotator quality parameters (ρ) that reflect their reliability in labeling.
• Uses a probabilistic framework where label assignments are inferred via variational inference, allowing for end-to-end learning of topics, labels, and annotator qualities.
• Supports partial labeling: annotators need not label all classes for a document, making the model practical for real-world crowd-sourcing settings.

Experimental results

Research questions

• RQ1 Can modeling label absence improve multi-label classification performance compared to models that only consider label presence?
• RQ2 Can a topic model jointly learn latent topics, label assignments, and annotator quality from noisy crowd labels?
• RQ3 How does the model perform when training data is limited and annotators are heterogeneous in quality?
• RQ4 Does the model remain robust in the presence of adversarial or low-quality annotators?
• RQ5 To what extent does increasing the number of topics or training data improve performance, especially under data scarcity?

Key findings

• ML-PA-LDA-C achieved average accuracy of 0.942 and micro-F1 of 0.669 on the Reuters dataset with 100% of the training data, outperforming SLDA despite using only noisy crowd labels.
• The model learned annotator qualities with an average Ann RMSE of 0.009 when 100% of the training data was used, showing high accuracy in quality estimation.
• Performance improved with more training data and a higher number of topics, consistent with Occam’s razor when data was scarce, but increased topic count helped with sufficient data.
• Even with 10% of the training data, ML-PA-LDA-C achieved 92.7% accuracy and 61.6% micro-F1, demonstrating strong performance with minimal supervision.
• In adversarial settings with 10% of annotators having ρ < 0.5, the model maintained high performance with 95.5% average accuracy and Ann RMSE of 0.002, showing robustness.
• The model's performance was consistently superior to SLDA and comparable to the non-crowd version (ML-PA-LDA), even when trained on noisy labels.