[Paper Review] GenericsKB: A Knowledge Base of Generic Statements

Sumithra Bhakthavatsalam, Chloe Anastasiades+1 more|arXiv (Cornell University)|May 2, 2020

Natural Language Processing Techniques19 references44 citations

TL;DR

GenericsKB is a large collection (3.5M+) of naturally occurring generic statements with topical metadata and a learned confidence, plus GenericsKB-Best (1M+ sentences) augmented with synthesized generics; it improves QA explanations and can boost performance on reasoning tasks compared to using larger generic corpora.

ABSTRACT

We present a new resource for the NLP community, namely a large (3.5M+ sentence) knowledge base of *generic statements*, e.g., "Trees remove carbon dioxide from the atmosphere", collected from multiple corpora. This is the first large resource to contain *naturally occurring* generic sentences, as opposed to extracted or crowdsourced triples, and thus is rich in high-quality, general, semantically complete statements. All GenericsKB sentences are annotated with their topical term, surrounding context (sentences), and a (learned) confidence. We also release GenericsKB-Best (1M+ sentences), containing the best-quality generics in GenericsKB augmented with selected, synthesized generics from WordNet and ConceptNet. In tests on two existing datasets requiring multihop reasoning (OBQA and QASC), we find using GenericsKB can result in higher scores and better explanations than using a much larger corpus. This demonstrates that GenericsKB can be a useful resource for NLP applications, as well as providing data for linguistic studies of generics and their semantics. GenericsKB is available at https://allenai.org/data/genericskb.

Motivation & Objective

• Provide a large, naturally occurring corpus of generic statements for NLP and linguistics.
• Annotate statements with topical metadata, surrounding context, and a learned confidence score.
• Release a high-quality subset (GenericsKB-Best) augmented with synthesized generics from WordNet and ConceptNet.
• Demonstrate utility of GenericsKB in downstream tasks like question answering and explanation generation.

Proposed method

• Collect sentences from 1.7B sentences across three corpora (Waterloo, SimpleWikipedia, ARC).
• Clean and filter noise using regex, length heuristics, and language detection.
• Identify standalone generics with 27 hand-authored lexico-syntactic rules.
• Score candidate generics with a BERT classifier trained on crowd-annotated judgments of usefulness as general truths.

Experimental results

Research questions

• RQ1 Can a large, naturally occurring generic statement corpus be built with high quality and contextual completeness?
• RQ2 Does GenericsKB improve performance or explanations in existing multihop reasoning tasks compared to larger generic corpora?
• RQ3 What is the quality and utility of GenericsKB for QA and explanation tasks?

Key findings

• The final GenericsKB contains 3,433,000 sentences with topical metadata, context, and a confidence score.
• GenericsKB-Best contains 1,020,868 generics (774,621 from GenericsKB plus 246,247 synthesized) augmented with WordNet/ConceptNet data.
• In OpenBookQA, GenericsKB-Best yields higher QA performance (0.678) than GenericsKB (0.632) and the QASC-17M baseline (0.660).
• GenericsKB-Best produces substantially better two-hop explanations on QASC (0.61 vs. 0.44 for QASC-17M; 0.79 vs. 0.66 in the other metric).
• Annotation/quality checks show about 85% agreement with usefulness criteria for GenericsKB-Best samples, indicating relatively low contextual leakage or vagueness.