[Paper Review] Studio Ousia's Quiz Bowl Question Answering System

Ikuya Yamada, Ryuji Tamaki+2 more|arXiv (Cornell University)|Mar 23, 2018

Topic Modeling4 citations

TL;DR

This paper presents Studio Ousia's winning system in the NIPS 2017 HCQA competition, which uses a hybrid approach combining two novel neural networks—Neural Quiz Solver and Neural Type Predictor—with conventional information retrieval models, all integrated via a supervised machine learning Answer Scorer. The system achieved 97% accuracy on full quiz bowl questions and defeated six top human quiz experts by a wide margin, scoring 425 points to the humans' 200.

ABSTRACT

In this chapter, we describe our question answering system, which was the winning system at the Human-Computer Question Answering (HCQA) Competition at the Thirty-first Annual Conference on Neural Information Processing Systems (NIPS). The competition requires participants to address a factoid question answering task referred to as quiz bowl. To address this task, we use two novel neural network models and combine these models with conventional information retrieval models using a supervised machine learning model. Our system achieved the best performance among the systems submitted in the competition and won a match against six top human quiz experts by a wide margin.

Motivation & Objective

• To develop a high-accuracy question answering system for the factoid quiz bowl task at the NIPS 2017 HCQA competition.
• To address the challenge of real-time answer prediction as questions are revealed word-by-word, requiring early and accurate entity prediction.
• To improve performance by integrating neural network models with traditional information retrieval techniques through a supervised ensemble model.
• To outperform both competing systems and elite human quiz players in a live, rule-compliant quiz bowl setting.

Proposed method

• The Neural Quiz Solver models the QA task as text classification, using averaged word and entity embeddings as input to predict the correct Wikipedia entity answer.
• The Neural Type Predictor uses a convolutional neural network (CNN) to predict coarse- and fine-grained entity types (e.g., person, author) of the answer, enhancing answer relevance.
• A supervised machine learning-based Answer Scorer combines features from both neural models and conventional IR models (using Wikipedia and Freebase) to rank candidate answers.
• The system employs a dynamic answer trigger: it outputs an answer only if the top candidate’s relevance score exceeds 0.6 and the question contains at least 15 words to avoid instability in short queries.
• The final model is trained on a curated dataset of 101,043 question-answer pairs, combining Protobowl and Iyyer et al. datasets after resolving answer mappings to Wikipedia titles.
• Model training uses early stopping on a 10% development set, and all components are optimized jointly within the ensemble framework.

Experimental results

Research questions

• RQ1 Can a hybrid neural and retrieval-based system achieve state-of-the-art performance on the quiz bowl QA task, where answers must be predicted incrementally as questions are revealed?
• RQ2 How effective are neural models that jointly encode words and Wikipedia entities in improving answer accuracy for factoid questions?
• RQ3 To what extent does predicting entity types (coarse and fine-grained) improve answer selection in a low-resource, incremental QA setting?
• RQ4 Can a machine learning ensemble combining neural and traditional IR models outperform both standalone neural models and human experts in a live quiz competition?

Key findings

• The system achieved 97% accuracy on full quiz bowl questions, demonstrating strong performance on complex, multi-sentence clues.
• With only the first sentence, the system achieved 56% accuracy, improving to 88% with three sentences, showing strong incremental learning capability.
• The integration of IR models (especially Wikipedia-based retrieval) significantly boosted performance, particularly for shorter questions, increasing accuracy from 89% to 95% when combined with neural features.
• The Neural Type Predictor achieved over 95% accuracy on coarse-grained entity type prediction for full questions, with fine-grained models performing well on Precision@1 but lower on F1 due to multi-type prediction challenges.
• In simulated pairwise matches, the system scored 1220 points against Acelove (60 points) and 1145 points against Lunit.io (105 points), demonstrating a decisive performance advantage.
• In a live match at the NIPS 2017 workshop, the system defeated a team of six elite human quiz experts, earning 425 points compared to the humans’ 200 points, confirming its superiority in real-world conditions.