[Paper Review] Learnability with Indirect Supervision Signals

Kaifu Wang, Ning Qiang+1 more|arXiv (Cornell University)|Jun 15, 2020

Machine Learning and Algorithms35 references4 citations

TL;DR

This paper introduces a unified theoretical framework for multiclass classification under indirect supervision, where labels are inferred from a noisy or partial signal with unknown, non-invertible, and instance-dependent transitions. The key contribution is the concept of 'separation'—a statistical distance measure between label distributions that characterizes learnability and generalization, enabling provable learning bounds under minimal assumptions about transition knowledge.

ABSTRACT

Learning from indirect supervision signals is important in real-world AI applications when, often, gold labels are missing or too costly. In this paper, we develop a unified theoretical framework for multi-class classification when the supervision is provided by a variable that contains nonzero mutual information with the gold label. The nature of this problem is determined by (i) the transition probability from the gold labels to the indirect supervision variables and (ii) the learner's prior knowledge about the transition. Our framework relaxes assumptions made in the literature, and supports learning with unknown, non-invertible and instance-dependent transitions. Our theory introduces a novel concept called \emph{separation}, which characterizes the learnability and generalization bounds. We also demonstrate the application of our framework via concrete novel results in a variety of learning scenarios such as learning with superset annotations and joint supervision signals.

Motivation & Objective

• To formalize a general framework for multiclass classification when gold labels are unavailable and only indirect supervision signals are available.
• To relax prior assumptions in the literature by allowing unknown, non-invertible, and instance-dependent transition probabilities between true labels and supervision signals.
• To identify the minimal prior knowledge about the transition process required for successful learning.
• To characterize the learnability of such problems through a new theoretical construct: separation.
• To provide generalization bounds and conditions under which consistent learning is possible despite imperfect supervision.

Proposed method

• The framework models the relationship between true labels and indirect supervision signals using a transition probability distribution, which is not assumed to be fully known.
• The concept of 'transition class' is introduced to represent the set of candidate transitions, enabling flexible prior knowledge representation.
• A new theoretical construct, 'separation', is defined as the statistical distance (e.g., total variation or KL divergence) between distributions of supervision signals induced by different true labels.
• Separation is used to quantify both consistency and identifiability of the learning problem, ensuring that different labels can be distinguished via the supervision signal.
• Theoretical bounds are derived using a decomposition into three components: complexity, consistency, and identifiability, leading to a unified generalization bound (Theorem 4.2).
• The framework is applied to concrete cases such as superset annotations and joint supervision, where separation is achieved via total variation or combined signal distributions.

Experimental results

Research questions

• RQ1 Under what conditions can a multiclass classifier be consistently learned when only indirect supervision signals are available?
• RQ2 What level of prior knowledge about the transition from true labels to supervision signals is sufficient for learnability?
• RQ3 How can the difficulty of learning from indirect supervision be formally quantified and bounded?
• RQ4 Can a unified theoretical framework be developed that subsumes existing settings like label noise and partial annotations?
• RQ5 How does the structure of the supervision signal (e.g., superset, joint signals) affect the learnability and generalization performance?

Key findings

• The paper establishes a unified generalization bound (Theorem 4.2) that decomposes learnability into complexity, consistency, and identifiability components.
• The concept of 'separation' is formally defined and shown to be a sufficient condition for identifiability and consistency, with a lower bound on separation ensuring generalization.
• For superset annotations, the paper shows that separation can be achieved via total variation distance between the distributions of possible annotations.
• In joint supervision settings, where multiple indirect signals are available, the framework proves that separation can be achieved by combining signals, even when individual signals are ambiguous.
• The paper demonstrates that if the support of supervision signals overlaps in a way that masks label information (e.g., when two noisy annotators produce symmetric signals), then separation can vanish, making learning impossible.
• A counterexample is provided showing that even if individual signals are separable, mixing them without distinguishing their sources can destroy all separation, rendering the signal uninformative.