[Paper Review] KGCleaner : Identifying and Correcting Errors Produced by Information Extraction Systems

Ankur Padia, Francis Ferraro+1 more|arXiv (Cornell University)|Aug 15, 2018

Topic Modeling14 references4 citations

TL;DR

KGCleaner is a multi-task framework that jointly identifies and corrects errors in knowledge base facts extracted by information extraction systems. It uses shallow neural networks to classify credibility and repair faulty relations, achieving a 30 F1-point gain on Wikidata and over twofold improvement in repair performance depending on dataset characteristics.

ABSTRACT

KG Cleaner is a framework to identify and correct errors in data produced and delivered by an information extraction system. These tasks have been understudied and KG Cleaner is the first to address both. We introduce a multi-task model that jointly learns to predict if an extracted relation is credible and repair it if not. We evaluate our approach and other models as instance of our framework on two collections: a Wikidata corpus of nearly 700K facts and 5M fact-relevant sentences and a collection of 30K facts from the 2015 TAC Knowledge Base Population task. For credibility classification, we find that parameter efficient, simple shallow neural networks can achieve an absolute performance gain of 30 F1 points on Wikidata and comparable performance on TAC. For the repair task, significant performance (at more than twice) gain can be obtained depending on the nature of the dataset and the models.

Motivation & Objective

• To address the underexplored problem of error identification and correction in knowledge base population from information extraction systems.
• To develop a unified framework that jointly learns credibility prediction and relation repair.
• To evaluate the framework on diverse, real-world datasets, including Wikidata and the 2015 TAC KBP benchmark.
• To demonstrate that simple, parameter-efficient models can outperform complex baselines in credibility classification.
• To quantify the gains in repair performance across different data distributions and model types.

Proposed method

• The framework employs a multi-task learning architecture that shares representations between credibility classification and relation repair tasks.
• It uses shallow, parameter-efficient neural networks to predict whether an extracted relation is credible.
• For repair, the model generates corrected relation triples when credibility is low, using sequence-to-sequence generation with attention mechanisms.
• The model is jointly trained on two datasets: a Wikidata corpus with ~700K facts and 5M relevant sentences, and a 30K-fact TAC KBP dataset.
• Training involves end-to-end optimization of both credibility prediction and repair generation objectives.
• The framework leverages contextual sentence features to improve both credibility assessment and repair accuracy.

Experimental results

Research questions

• RQ1 Can a multi-task learning framework effectively identify and correct errors in extracted knowledge base facts?
• RQ2 How do shallow neural networks compare to more complex models in credibility classification for knowledge base facts?
• RQ3 To what extent can joint learning of credibility and repair improve overall performance?
• RQ4 How does performance vary across different datasets and error types?
• RQ5 What is the impact of model complexity and parameter efficiency on credibility and repair tasks?

Key findings

• Shallow neural networks achieved an absolute 30 F1-point improvement in credibility classification on the Wikidata dataset compared to baseline models.
• On the TAC KBP dataset, the same models achieved comparable performance to more complex architectures, demonstrating robustness across datasets.
• The repair task saw performance gains of more than twofold depending on the dataset and model configuration.
• Joint learning of credibility and repair led to significant improvements, especially in complex or noisy data distributions.
• The framework demonstrated strong generalization across diverse knowledge base populations, including Wikidata and TAC KBP.
• Parameter efficiency was maintained without sacrificing performance, making the approach scalable and practical for real-world deployment.