[Paper Review] Joint training for open-domain extraction on the web: exploiting overlap when supervision is limited

Rahul Gupta, Sunita Sarawagi|arXiv (Cornell University)|Feb 9, 2011

Topic Modeling28 references21 citations

TL;DR

This paper proposes an agreement-based learning framework that jointly trains open-domain extraction models across multiple web sources by leveraging overlapping text segments to improve performance under limited supervision. By biasing models to agree on shared content regions and using a partitioning scheme for efficient training, the method achieves significant accuracy gains over uncoupled training and existing approaches like collective inference and multi-view learning.

ABSTRACT

We consider the problem of jointly training structured mod-els for extraction from multiple web sources whose records enjoy partial content overlap. This has important applica-tions in open-domain extraction, e.g. a user materializing a table of interest from multiple relevant unstructured sources; or a site like Freebase augmenting an incomplete relation by extracting more rows from web sources. Such applications require extraction over arbitrary domains, so one cannot use a pre-trained extractor or demand a huge labeled dataset. We propose to overcome this lack of supervision by using content overlap across the related web sources. Existing methods of exploiting overlap have been developed under settings that do not generalize easily to the scale and diver-sity of overlap seen on Web sources. We present an agreement-based learning framework that jointly trains the models by biasing them to agree on the agreement regions, i.e. shared text segments. We present alternatives within our framework to trade-off tractability, robustness to noise, and extent of agreement enforced; and propose a scheme of partitioning agreement regions that leads to efficient training while maximizing overall accuracy. Further, we present a principled scheme to discover low-noise agreement regions in unlabeled data across multiple sources. Through extensive experiments over 58 different extrac-tion domains, we establish that our framework provides sig-nificant boosts over uncoupled training, and scores over al-ternatives such as collective inference, staged training, and multi-view learning.

Motivation & Objective

• To address the challenge of low-resource, open-domain information extraction from diverse, unlabeled web sources with partial content overlap.
• To overcome the limitations of existing methods that fail to scale or generalize across the diversity of real-world web data.
• To develop a joint training framework that exploits overlap across sources without requiring extensive labeled data.
• To discover low-noise agreement regions in unlabeled data to improve model robustness and training efficiency.
• To achieve higher extraction accuracy than uncoupled training, collective inference, staged training, and multi-view learning.

Proposed method

• The framework uses an agreement-based learning approach that biases multiple extraction models to produce consistent predictions on shared text segments across sources.
• It introduces a partitioning strategy for agreement regions to balance training efficiency and performance, enabling scalable joint optimization.
• The method includes configurable trade-offs between tractability, noise robustness, and the extent of enforced agreement across models.
• A principled scheme is proposed to automatically identify low-noise agreement regions in unlabeled data, reducing reliance on noisy supervision.
• The framework supports joint training of models without requiring pre-trained extractors or large labeled datasets.
• It enables end-to-end learning that maximizes agreement on overlapping content while maintaining generalization across diverse domains.

Experimental results

Research questions

• RQ1 Can joint training of extraction models across multiple overlapping web sources significantly improve performance when labeled data is scarce?
• RQ2 How can agreement regions be effectively discovered and utilized in unlabeled, diverse web data to guide joint training?
• RQ3 What trade-offs exist between training efficiency, noise robustness, and agreement enforcement in joint extraction?
• RQ4 How does the proposed framework compare in accuracy and scalability to collective inference, staged training, and multi-view learning?
• RQ5 To what extent can overlap across sources be exploited to reduce dependency on large labeled datasets in open-domain extraction?

Key findings

• The proposed framework achieves significant accuracy improvements over uncoupled training across 58 different extraction domains.
• It outperforms alternative methods such as collective inference, staged training, and multi-view learning in both accuracy and robustness.
• The framework enables effective joint training even with limited supervision by leveraging content overlap across sources.
• The partitioning of agreement regions leads to efficient training while maximizing overall extraction accuracy.
• The scheme for discovering low-noise agreement regions in unlabeled data enhances model performance and reduces sensitivity to noisy annotations.
• The method demonstrates strong generalization across diverse, real-world web sources without requiring pre-trained models or large-scale labeled data.