[Paper Review] Open Question Answering with Weakly Supervised Embedding Models

Antoine Bordes, Jason Weston+1 more|arXiv (Cornell University)|Apr 16, 2014

Topic ModelingComputer Science24 references86 citations

TL;DR

This paper proposes a weakly supervised neural embedding model for open-domain question answering that maps questions and answers into a shared vector space without requiring hand-crafted grammars or lexicons. By leveraging automatically generated question-triple pairs and paraphrase data for weak supervision, and introducing a fine-tuning procedure for similarity matrix optimization, the model achieves state-of-the-art performance on the WikiAnswers+ReVerb dataset, outperforming prior methods like Paralex even in zero-shot settings on WebQuestions.

ABSTRACT

Building computers able to answer questions on any subject is a long standing goal of artificial intelligence. Promising progress has recently been achieved by methods that learn to map questions to logical forms or database queries. Such approaches can be effective but at the cost of either large amounts of human-labeled data or by defining lexicons and grammars tailored by practitioners. In this paper, we instead take the radical approach of learning to map questions to vectorial feature representations. By mapping answers into the same space one can query any knowledge base independent of its schema, without requiring any grammar or lexicon. Our method is trained with a new optimization procedure combining stochastic gradient descent followed by a fine-tuning step using the weak supervision provided by blending automatically and collaboratively generated resources. We empirically demonstrate that our model can capture meaningful signals from its noisy supervision leading to major improvements over paralex, the only existing method able to be trained on similar weakly labeled data.

Motivation & Objective

• To enable open-domain question answering without requiring human-annotated (question, answer) pairs or schema-specific grammars.
• To learn meaningful vectorial representations of questions and KB triples using only weak supervision from automatically generated data and paraphrase signals.
• To improve model generalization and performance through a novel fine-tuning procedure for embedding similarity matrices.
• To evaluate the model’s zero-shot transfer capability on a different KB and question set (WebQuestions) without retraining.

Proposed method

• The model learns dense vector embeddings for words and KB triples such that question embeddings are close to their corresponding answer embeddings in the shared space.
• It uses a contrastive loss objective to push positive (question, triple) pairs closer and negative pairs farther apart in the embedding space.
• Weak supervision is derived from automatically generated question-triple pairs using a rule-based template system applied to KB triples.
• Additional weak supervision comes from a dataset of paraphrased questions with no associated answers, which helps the model learn semantic equivalence.
• A fine-tuning step optimizes a matrix parameterizing the similarity function in the embedding space to improve performance.
• String matching is applied during inference to restrict candidate triples to a manageable set (~10k per question), improving efficiency and recall.

Experimental results

Research questions

• RQ1 Can a neural embedding model learn effective question-answer representations using only weak supervision from automatically generated data?
• RQ2 How well can such a model generalize to a different KB and question set without retraining?
• RQ3 Can a fine-tuning procedure for the similarity matrix in the embedding space improve performance on open-domain QA?
• RQ4 To what extent can the model capture semantic relationships like synonymy and hypernymy without explicit supervision?

Key findings

• The model achieves an F1 score of 0.076 on the WebQuestions test set when using both fine-tuning and string matching, significantly outperforming the baseline embedding model (F1 0.025).
• The fine-tuned model improves Top-1 accuracy from 0.032 to 0.094 and Top-10 from 0.106 to 0.270 when combined with string matching.
• The model learns meaningful semantic relationships, such as synonymy (e.g., 'radiation' and 'x-ray.e') and hyponymy (e.g., 'iphone' and 'smartphone.e'), through weak supervision.
• Even without retraining, the model generalizes to the WebQuestions dataset, achieving a Top-1 accuracy of 0.094 on 1,538 questions, demonstrating strong zero-shot transfer capability.
• The use of string matching during inference reduces the candidate set from 14 million to around 10,000 triples per question, significantly improving efficiency and recall.