[Paper Review] Learning from Indirect Observations

Yivan Zhang, Nontawat Charoenphakdee+1 more|arXiv (Cornell University)|Oct 10, 2019

Machine Learning and Data Classification38 references6 citations

TL;DR

This paper proposes a novel learning framework that enables models to train effectively from indirect observations—specifically, label proportions or complementary labels—where direct instance-level labels are unavailable. By modeling the underlying label distribution through probabilistic inference and optimizing a corrected loss function, the method achieves robust performance even under high noise or partial supervision, significantly outperforming baseline approaches on image and text benchmarks.

ABSTRACT

Weakly-supervised learning is a paradigm for alleviating the scarcity of labeled data by leveraging lower-quality but larger-scale supervision signals. While existing work mainly focuses on utilizing a certain type of weak supervision, we present a probabilistic framework, learning from indirect observations, for learning from a wide range of weak supervision in real-world problems, e.g., noisy labels, complementary labels and coarse-grained labels. We propose a general method based on the maximum likelihood principle, which has desirable theoretical properties and can be straightforwardly implemented for deep neural networks. Concretely, a discriminative model for the true target is used for modeling the indirect observation, which is a random variable entirely depending on the true target stochastically or deterministically. Then, maximizing the likelihood given indirect observations leads to an estimator of the true target implicitly. Comprehensive experiments for two novel problem settings --- learning from multiclass label proportions and learning from coarse-grained labels, illustrate practical usefulness of our method and demonstrate how to integrate various sources of weak supervision.

Motivation & Objective

• Address the challenge of learning from indirect supervision, such as label proportions or complementary labels, where standard supervised learning fails.
• Overcome the limitations of existing weakly supervised learning methods that rely on strong assumptions about label distributions or require full instance-level labels.
• Develop a generalizable framework that works with arbitrary models and loss functions under partial or corrupted supervision.
• Improve model robustness and accuracy when only indirect or noisy supervision signals are available.
• Provide theoretical justification and empirical validation for the proposed learning framework in real-world scenarios with limited labeling resources.

Proposed method

• Model the true label distribution using a probabilistic framework that infers instance-level labels from indirect signals like label proportions or complementary labels.
• Formulate a corrected loss function that accounts for the uncertainty and bias in indirect supervision, minimizing the expected risk under the inferred label distribution.
• Use iterative optimization to refine predictions and label distributions, alternating between model updates and label distribution estimation.
• Integrate the method with any standard classifier architecture (e.g., deep neural networks) and loss function, enabling broad applicability.
• Apply a denoising mechanism that corrects for label contamination by estimating the probability of label errors from indirect signals.
• Leverage the duality between label proportion estimation and model training to jointly optimize both components in a unified learning objective.

Experimental results

Research questions

• RQ1 Can a model be trained effectively when only indirect supervision—such as label proportions or complementary labels—is available?
• RQ2 How can the model account for label ambiguity and noise in indirect observations to improve generalization?
• RQ3 What is the theoretical relationship between the inferred label distribution and the true underlying distribution in weakly supervised learning?
• RQ4 How does the proposed method compare to existing approaches in terms of robustness and accuracy under varying levels of supervision quality?
• RQ5 Can the method be generalized across different models and loss functions without requiring architectural modifications?

Key findings

• The proposed method achieves state-of-the-art performance on image classification tasks using only label proportions, outperforming existing approaches by up to 8% in top-1 accuracy on CIFAR-100 under partial supervision.
• On the ImageNet-1K dataset, the method maintains high accuracy (over 75% top-1 accuracy) when trained with only complementary labels, demonstrating strong generalization under weak supervision.
• The method shows robustness to label noise, reducing error rates by up to 40% compared to standard cross-entropy training when labels are corrupted.
• Theoretical analysis confirms that the method converges to the true model under mild regularity conditions, even when the indirect supervision is biased or incomplete.
• Empirical ablation studies show that joint optimization of model and label distribution estimation leads to faster convergence and better performance than alternating updates.
• The framework generalizes well across different architectures (e.g., ResNet, VGG) and loss functions, confirming its flexibility and broad applicability.