[Paper Review] Using Multi-Label Classification for Improved Question Answering

Ricardo Usbeck, Michael M. Hoffmann+3 more|arXiv (Cornell University)|Oct 24, 2017

Topic Modeling25 references3 citations

TL;DR

This paper proposes a multi-label classification-based metasystem that selects the best question answering (QA) system from six existing RDF-based QA engines using 14 novel question features. By training a PSt classifier on 100 QALD-6 questions, the metasystem achieves a 14.1% improvement in F-measure (0.78) over the best single system, demonstrating that feature-engineered meta-learning can effectively combine diverse QA approaches for superior performance despite limited training data.

ABSTRACT

A plethora of diverse approaches for question answering over RDF data have been developed in recent years. While the accuracy of these systems has increased significantly over time, most systems still focus on particular types of questions or particular challenges in question answering. What is a curse for single systems is a blessing for the combination of these systems. We show in this paper how machine learning techniques can be applied to create a more accurate question answering metasystem by reusing existing systems. In particular, we develop a multi-label classification-based metasystem for question answering over 6 existing systems using an innovative set of 14 question features. The metasystem outperforms the best single system by 14% F-measure on the recent QALD-6 benchmark. Furthermore, we analyzed the influence and correlation of the underlying features on the metasystem quality.

Motivation & Objective

• To address the challenge of selecting the most accurate QA system among multiple specialized systems for RDF data.
• To improve overall question answering performance by combining outputs of existing QA systems through a meta-learning approach.
• To investigate which question features most influence system selection accuracy in a multi-label classification framework.
• To evaluate the robustness and generalization of the metasystem across different training and test data configurations.
• To identify limitations in current QA systems, particularly for complex questions requiring solution modifiers.

Proposed method

• The metasystem uses a multi-label classification framework to predict which of six existing QA systems is most likely to answer a given question correctly.
• It extracts 14 question features, including question word (QW), number of tokens (#T), location (Loc), query resource type (QRT), and person (Pers), to characterize input questions.
• A PSt classifier is selected as the best-performing model based on F1-score evaluation using both leave-one-out and full training setups.
• Feature importance is analyzed by training the PSt classifier on various combinations of features to identify the optimal subset for performance.
• The system is trained and tested on the QALD-6 benchmark, using 100 questions for both training and evaluation in the primary setup.
• Performance is evaluated using F1-measure, with comparisons to individual QA systems and an idealized optimal selection baseline.

Experimental results

Research questions

• RQ1 Can a meta-system that selects from multiple existing QA systems outperform the best single system on RDF question answering?
• RQ2 Which combination of question features most effectively predicts the best-performing QA system for a given question?
• RQ3 How does the performance of the metasystem vary between leave-one-out and full training setups, and what does this imply about overfitting?
• RQ4 To what extent do feature combinations improve or degrade the metasystem’s F1-score, and is there a risk of overfitting with limited training data?
• RQ5 What are the persistent weaknesses of current QA systems, and how do they affect the metasystem’s ability to generalize?

Key findings

• The metasystem achieves an F1-measure of 0.78, representing a 14.1% improvement over the best single QA system (F1 = 0.68) on the QALD-6 benchmark.
• The optimal feature set for the PSt classifier includes #T (number of tokens), Loc (location), QW (question word), and QRT (query resource type), achieving 0.78 F1.
• Adding all 14 features reduced performance to 0.76 F1, indicating that additional features introduced noise rather than signal.
• The leave-one-out setup showed lower performance (F1 ≈ 0.63–0.69) than the full training setup (F1 = 0.76), suggesting potential overfitting in the smaller setup.
• The metasystem still falls short of the ideal selection baseline (F1 = 0.89), highlighting the impact of limited training data (only 100 questions).
• Questions requiring solution modifiers (e.g., 9, 88, 17, 28, 33, 36, 49) remain particularly challenging for all systems, including the metasystem.