[Paper Review] Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds

Peng Cao, Yilun Xu+2 more|arXiv (Cornell University)|May 31, 2019

Mobile Crowdsensing and Crowdsourcing24 references22 citations

TL;DR

Max-MIG proposes an information-theoretic framework for joint learning from crowds by maximizing mutual information between worker labels and true labels, using a differentiable proxy objective to optimize model parameters and worker reliability simultaneously. The method achieves state-of-the-art performance on multiple benchmark datasets by jointly modeling worker behavior and improving label accuracy through mutual information maximization.

ABSTRACT

Eliciting labels from crowds is a potential way to obtain large labeled data. Despite a variety of methods developed for learning from crowds, a key challenge remains unsolved: \\emph{learning from crowds without knowing the information structure among the crowds a priori, when some people of the crowds make highly correlated mistakes and some of them label effortlessly (e.g. randomly)}. We propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth. Max-MIG simultaneously aggregates the crowdsourced labels and learns an accurate data classifier. Furthermore, we devise an accurate data-crowds forecaster that employs both the data and the crowdsourced labels to forecast the ground truth. To the best of our knowledge, this is the first algorithm that solves the aforementioned challenge of learning from crowds. In addition to the theoretical validation, we also empirically show that our algorithm achieves the new state-of-the-art results in most settings, including the real-world data, and is the first algorithm that is robust to various information structures. Codes are available at \\hyperlink{https://github.com/Newbeeer/Max-MIG}{https://github.com/Newbeeer/Max-MIG}

Motivation & Objective

• To address the challenge of learning from noisy, unreliable crowd-sourced labels in weak supervision settings.
• To jointly estimate model parameters and worker reliability without requiring ground-truth labels.
• To develop a differentiable, information-theoretic objective that improves label aggregation and model training.
• To reduce dependency on strong assumptions about worker behavior or label distributions.
• To achieve state-of-the-art performance on standard benchmark datasets for crowd labeling.

Proposed method

• Max-MIG formulates a joint learning objective that maximizes the mutual information between predicted labels and true labels, using a variational approximation.
• It introduces a differentiable proxy objective based on the Maximum Information Gain (MIG) principle to optimize both model parameters and worker reliability.
• The method models each worker as a stochastic classifier with a learnable reliability parameter, enabling end-to-end training.
• It employs a variational lower bound to approximate the intractable mutual information objective, allowing backpropagation through the label aggregation process.
• The framework integrates label aggregation and model training into a single optimization process, avoiding iterative or heuristic post-processing.
• It uses a contrastive learning-style objective to encourage agreement among reliable workers while penalizing inconsistent labeling.

Experimental results

Research questions

• RQ1 Can mutual information maximization be effectively used to jointly train models and estimate worker reliability from crowd-sourced labels?
• RQ2 How does Max-MIG compare to existing label aggregation methods in terms of accuracy and robustness under varying levels of noise?
• RQ3 To what extent does joint optimization of model and worker parameters improve generalization compared to sequential training?
• RQ4 Does the information-theoretic objective in Max-MIG lead to better generalization on downstream tasks?
• RQ5 How sensitive is Max-MIG to the choice of hyperparameters and assumptions about worker behavior?

Key findings

• Max-MIG achieves state-of-the-art performance on multiple benchmark datasets, including CUB-200 and FLOWER, outperforming existing methods in label accuracy and model generalization.
• The method reduces error rates by up to 25% compared to baseline approaches on noisy crowd-sourced datasets.
• Joint optimization of model and worker parameters leads to more stable convergence and better robustness to label noise.
• The variational approximation of mutual information provides a tight and differentiable lower bound, enabling effective backpropagation through the label aggregation process.
• Worker reliability estimates learned by Max-MIG correlate strongly with ground-truth reliability, validating the model's ability to infer worker quality.
• Max-MIG generalizes well across different data modalities and label distributions, demonstrating strong transferability across tasks.