[Paper Review] ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers

Haitian Sun, William W. Cohen+1 more|arXiv (Cornell University)|Oct 13, 2021

Topic Modeling4 citations

TL;DR

This paper introduces ConditionalQA, a complex reading comprehension dataset featuring questions with conditional answers—answers that are valid only under specific, explicitly stated conditions. The dataset challenges models to perform multi-hop reasoning over long, logically complex documents, with state-of-the-art models achieving only 64.9% accuracy on yes/no questions and 25.2% exact match on extractive answers, highlighting significant gaps in current models' ability to reason with conditional logic.

ABSTRACT

We describe a Question Answering (QA) dataset that contains complex questions with conditional answers, i.e. the answers are only applicable when certain conditions apply. We call this dataset ConditionalQA. In addition to conditional answers, the dataset also features: (1) long context documents with information that is related in logically complex ways; (2) multi-hop questions that require compositional logical reasoning; (3) a combination of extractive questions, yes/no questions, questions with multiple answers, and not-answerable questions; (4) questions asked without knowing the answers. We show that ConditionalQA is challenging for many of the existing QA models, especially in selecting answer conditions. We believe that this dataset will motivate further research in answering complex questions over long documents. Data and leaderboard are publicly available at \url{https://github.com/haitian-sun/ConditionalQA}.

Motivation & Objective

• To address the gap in existing reading comprehension datasets, which typically feature deterministic answers, by introducing a dataset with conditional answers that depend on specific context-dependent conditions.
• To create a benchmark that tests models' ability to perform multi-hop reasoning over long, logically structured documents with complex internal dependencies.
• To evaluate the performance of current state-of-the-art QA models on questions that require identifying both answers and their associated conditions, revealing limitations in reasoning and condition selection.
• To decouple the question-asking and answering processes during annotation, ensuring questions are not biased by prior knowledge of the answers, thus improving realism and diversity.

Proposed method

• The dataset is constructed from real public policy documents in the UK, focusing on social benefits like Funeral Expense Payment, which involve complex eligibility criteria with conditional logic.
• Each example includes a user question, a scenario description, a long policy document, and human-annotated supporting evidence, with answers explicitly paired with their required conditions.
• Questions are diverse: extractive, yes/no, multi-answer, and not-answerable, reflecting real-world ambiguity and incompleteness in user queries.
• The annotation process separates question creation from answer selection, ensuring questions are asked without knowing the answers, increasing realism and reducing bias.
• Models are evaluated on both answer accuracy and condition accuracy, with joint evaluation of answers and conditions to measure holistic reasoning performance.
• Baseline models include retrieval-augmented encoders (e.g., ETC-pipeline, DocHopper) and a generative model (FiD), with performance measured via exact match (EM) and F1 on answers and conditions.

Experimental results

Research questions

• RQ1 Can existing question-answering models correctly identify answers that are only valid under specific, explicitly stated conditions in complex policy documents?
• RQ2 How well do current models generalize to questions that require multi-hop reasoning across multiple logical dependencies in long, structured documents?
• RQ3 To what extent do models fail to select correct conditions for answers, and what are the primary error patterns in conditional answer prediction?
• RQ4 Does the performance of models degrade significantly when jointly predicting answers and conditions, compared to predicting answers alone?
• RQ5 Can threshold tuning on confidence scores improve the prediction of conditional answers without introducing excessive false positives?

Key findings

• The best-performing model achieved only 64.9% accuracy on yes/no questions—marginally better than the majority baseline of 62.2% that always predicts 'yes'.
• On extractive questions, the best model achieved a 25.2% exact match (EM), indicating significant difficulty in identifying correct answer spans.
• When jointly evaluating answers and conditions, the best model performance dropped to 49.1% EM on yes/no questions and 22.5% EM on extractive questions, with the best results obtained when no conditions were predicted.
• The performance on questions with at least one conditional answer dropped by over 90% when conditions were jointly predicted, highlighting the extreme difficulty of condition selection.
• Error analysis revealed that the most common error was predicting only a subset of correct answers, especially due to models favoring single-answer outputs even when multiple answers are valid.
• Threshold tuning on condition confidence scores showed marginal improvement on conditional-answer-only subsets but caused a sharp decline in overall performance due to false positive conditions.