[Paper Review] Large-scale Simple Question Answering with Memory Networks

Antoine Bordes, Nicolas Usunier+2 more|arXiv (Cornell University)|Jun 5, 2015

Topic Modeling18 references562 citations

TL;DR

This paper introduces a large-scale simple question answering system using Memory Networks (MemNNs) trained on a new 100k-question dataset called SimpleQuestions, demonstrating state-of-the-art performance on WebQuestions (42.2% F1) and successful transfer learning to Reverb, achieving 67% accuracy without retraining. The approach leverages joint learning across multiple data sources and efficient retrieval via distributed representations in a shared vector space.

ABSTRACT

Training large-scale question answering systems is complicated because training sources usually cover a small portion of the range of possible questions. This paper studies the impact of multitask and transfer learning for simple question answering; a setting for which the reasoning required to answer is quite easy, as long as one can retrieve the correct evidence given a question, which can be difficult in large-scale conditions. To this end, we introduce a new dataset of 100k questions that we use in conjunction with existing benchmarks. We conduct our study within the framework of Memory Networks (Weston et al., 2015) because this perspective allows us to eventually scale up to more complex reasoning, and show that Memory Networks can be successfully trained to achieve excellent performance.

Motivation & Objective

• To address the lack of large-scale, diverse training data for simple question answering (QA) systems.
• To investigate the effectiveness of multitask and transfer learning in improving generalization across diverse QA datasets.
• To evaluate whether Memory Networks can scale effectively to large knowledge bases while maintaining high performance on simple QA tasks.
• To introduce a new benchmark dataset, SimpleQuestions, with 100k human-annotated questions and associated Freebase facts.
• To demonstrate that MemNNs can integrate new knowledge (e.g., Reverb facts) without retraining, enabling transfer learning in QA.

Proposed method

• The authors use Memory Networks (MemNNs) to embed questions, facts, and answers into a shared vector space for similarity-based retrieval.
• They train the model using a combination of real and synthetic (paraphrased) questions from multiple datasets, including WebQuestions, SimpleQuestions, and Reverb.
• The model employs cosine similarity between question embeddings and memory facts to rank candidate answers.
• The knowledge base is preprocessed to remove mediator nodes, enabling direct single-fact lookup and simplifying inference.
• Distant supervision is applied by grouping multiple facts with the same subject and relation to improve training efficiency and generalization.
• Transfer learning is evaluated by adding Reverb facts to the memory without retraining, testing the model’s ability to generalize to new entities and relations.

Experimental results

Research questions

• RQ1 Can a Memory Network-based QA system achieve state-of-the-art performance on large-scale simple QA benchmarks like WebQuestions?
• RQ2 Does joint training on multiple QA datasets improve generalization and performance across different question patterns?
• RQ3 Can a pre-trained MemNN system effectively answer questions from a new knowledge base (e.g., Reverb) without retraining?
• RQ4 How does data diversity and paraphrasing affect model robustness to syntactic and lexical variation in questions?
• RQ5 To what extent does the structure of the knowledge base (e.g., removal of mediator nodes) impact the performance of simple QA systems?

Key findings

• The proposed MemNN model achieves a test F1-score of 42.2% on the WebQuestions benchmark, outperforming prior state-of-the-art methods.
• On the new SimpleQuestions dataset, the model reaches 62–63% accuracy, significantly outperforming baseline retrieval methods.
• Transfer learning to the Reverb dataset yields 67% accuracy without retraining, surpassing the original Reverb paper’s 54% and nearing the 73% SOTA.
• Joint training on both WebQuestions and SimpleQuestions improves performance on both datasets, with no negative interaction, indicating effective multitask learning.
• Paraphrased questions significantly improve performance on Reverb due to its high lexical and syntactic variability, but have less impact on simpler, more structured datasets.
• The model demonstrates robustness to larger knowledge bases (e.g., FB5M), maintaining performance even when the KB contains more entities than necessary.