[Paper Review] A Knowledge Hunting Framework for Common Sense Reasoning

Ali Emami, Noelia De La Cruz+3 more|arXiv (Cornell University)|Jan 1, 2018

Topic Modeling21 references4 citations

TL;DR

This paper proposes a knowledge-hunting framework that uses web search to dynamically retrieve and classify common-sense knowledge for resolving pronoun coreference and causal reasoning tasks. By generating targeted queries from problem structures, extracting relevant evidence from search results, and weighting it for plausibility, the system achieves an F1 score of 0.51 on the Winograd Schema Challenge—surpassing the previous state-of-the-art by 0.21 and marking the first system to exceed 0.5 F1.

ABSTRACT

We introduce an automatic system that achieves state-of-the-art results on the Winograd Schema Challenge (WSC), a common sense reasoning task that requires diverse, complex forms of inference and knowledge. Our method uses a knowledge hunting module to gather text from the web, which serves as evidence for candidate problem resolutions. Given an input problem, our system generates relevant queries to send to a search engine, then extracts and classifies knowledge from the returned results and weighs them to make a resolution. Our approach improves F1 performance on the full WSC by 0.21 over the previous best and represents the first system to exceed 0.5 F1. We further demonstrate that the approach is competitive on the Choice of Plausible Alternatives (COPA) task, which suggests that it is generally applicable.

Motivation & Objective

• Address the challenge of common-sense reasoning in NLP, particularly in resolving ambiguous pronouns and causal relationships that require world knowledge beyond syntax.
• Overcome the limitations of static knowledge bases and neural models that fail to capture the breadth and diversity of common-sense knowledge.
• Develop a general-purpose system capable of solving all Winograd Schema Challenge instances—not just restricted subsets—by dynamically sourcing knowledge from the web.
• Demonstrate the generalizability of the approach to other common-sense reasoning tasks, such as the Choice of Plausible Alternatives (COPA) task.
• Improve performance on the Winograd Schema Challenge beyond previous methods by leveraging real-world textual evidence through information retrieval.

Proposed method

• Parse input problems into a semantic representation schema that captures key entities, predicates, and relations.
• Generate targeted search queries from the schema, using syntactic and semantic cues to improve relevance.
• Use a search engine to retrieve web snippets containing candidate knowledge, then parse and filter results using NLP tools.
• Classify and score retrieved evidence based on syntactic and semantic alignment with candidate resolutions, using heuristics for plausibility.
• Apply a weighted voting mechanism to combine evidence strength across multiple snippets, selecting the most supported antecedent or alternative.
• Adapt the framework for COPA by generating dual query sets for each candidate, enforcing temporal or causal ordering in retrieved results.

Experimental results

Research questions

• RQ1 Can dynamic web-based knowledge retrieval outperform static knowledge bases and neural models in common-sense reasoning tasks?
• RQ2 To what extent can real-world textual evidence from the web resolve ambiguous pronoun references in the Winograd Schema Challenge?
• RQ3 Does the knowledge-hunting framework generalize effectively to other common-sense reasoning tasks like COPA?
• RQ4 How do query generation quality and result filtering impact the reliability and performance of the system?
• RQ5 Can information retrieval techniques effectively model long-tailed, implicit, or rare common-sense knowledge not captured in existing corpora?

Key findings

• The proposed knowledge-hunting framework achieves an F1 score of 0.51 on the full Winograd Schema Challenge, representing a 0.21 improvement over the previous state-of-the-art.
• It is the first system to exceed an F1 score of 0.5 on the Winograd Schema Challenge, demonstrating a significant leap in performance.
• The system performs competitively on the COPA task, achieving 66.2% accuracy on the test set with a simplified AGQ method, showing generalization potential.
• The use of WordNet synonyms in query expansion did not improve performance on COPA and may have introduced noise, suggesting that synonym injection requires careful filtering.
• Misleading results were primarily caused by incomplete queries (e.g., missing context like "couldn't lift"), parsing errors (e.g., mislabeling "lift" as a verb), and noisy or incomplete snippets.
• The framework demonstrates that general world knowledge about plausibility is implicitly encoded in natural language text and can be effectively mined for reasoning tasks.