[Paper Review] Diversify and Disambiguate: Learning From Underspecified Data

Yoonho Lee, Huaxiu Yao+1 more|arXiv (Cornell University)|Feb 7, 2022

Machine Learning and Data Classification4 citations

TL;DR

DivDis is a two-stage framework that first learns a diverse set of hypotheses from underspecified data by training multiple heads on a shared backbone to disagree on unlabeled target data, then disambiguates by selecting the best function using minimal target labels. It achieves over 15% improvement in worst-group accuracy on the Waterbirds benchmark and outperforms semi-supervised methods on Camelyon17, demonstrating robustness to severe distribution shifts and underspecification.

ABSTRACT

Many datasets are underspecified: there exist multiple equally viable solutions to a given task. Underspecification can be problematic for methods that learn a single hypothesis because different functions that achieve low training loss can focus on different predictive features and thus produce widely varying predictions on out-of-distribution data. We propose DivDis, a simple two-stage framework that first learns a diverse collection of hypotheses for a task by leveraging unlabeled data from the test distribution. We then disambiguate by selecting one of the discovered hypotheses using minimal additional supervision, in the form of additional labels or inspection of function visualization. We demonstrate the ability of DivDis to find hypotheses that use robust features in image classification and natural language processing problems with underspecification.

Motivation & Objective

• To address the challenge of distribution shift in machine learning where training data is underspecified, allowing multiple plausible hypotheses to explain the data.
• To overcome the limitations of standard ERM and robustness methods that select only one function, even when multiple valid explanations exist.
• To improve generalization under subpopulation shift and domain generalization by discovering diverse, complementary hypotheses before selecting the best one.
• To enable sample-efficient adaptation to target domains using only a few labeled examples by leveraging disagreement on unlabeled target data.
• To provide a scalable, end-to-end framework that discovers meaningful, disentangled features without requiring spurious attribute annotations.

Proposed method

• DivDis uses a multi-headed neural network with a shared backbone, where each head represents a different hypothesis.
• During the Diversify stage, all heads are trained to minimize source loss on labeled data while being encouraged to make diverse predictions on an unlabeled target dataset.
• Diversity is enforced via a disagreement loss that maximizes prediction disagreement across heads on the target data, promoting exploration of alternative functions.
• In the Disambiguate stage, one head is selected based on labels from a small, informative subset of target data, using uncertainty or disagreement metrics to guide selection.
• The framework operates in two stages: first, diverse hypotheses are discovered without target labels; second, minimal supervision resolves ambiguity.
• The method scales to large-scale vision and text benchmarks and does not require prior knowledge of spurious features or causal structure.

Experimental results

Research questions

• RQ1 Can a learning framework discover multiple plausible hypotheses from underspecified data to improve robustness under distribution shift?
• RQ2 How can diversity among hypotheses be effectively encouraged during training without relying on labeled target data?
• RQ3 To what extent can minimal target labeling improve performance when multiple hypotheses are available?
• RQ4 Does the framework outperform standard ERM and existing robustness methods in settings with high underspecification?
• RQ5 Can the framework automatically identify and leverage complementary, disentangled features without explicit supervision?

Key findings

• DivDis achieves over 15% absolute improvement in worst-group accuracy on the Waterbirds benchmark without using any spurious attribute annotations.
• On the Camelyon17 dataset, DivDis outperforms existing semi-supervised learning methods under domain shift.
• With only 4 target domain labels, DivDis surpasses fine-tuning baselines that use 128 labels in completely correlated spurious attribute settings.
• The method discovers meaningful, complementary features automatically, as shown in visualization results for image and text tasks.
• DivDis scales effectively to large-scale benchmarks, including ImageNet and text classification, and maintains strong performance across diverse distribution shift settings.
• The framework demonstrates that diversity in hypotheses, combined with minimal disambiguation, leads to significantly improved robustness compared to single-function models.