[Paper Review] QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines

Valentina Pyatkin, Ayal Klein+2 more|arXiv (Cornell University)|Oct 6, 2020

Topic Modeling39 references4 citations

TL;DR

This paper introduces QADiscourse, a novel framework that represents discourse relations as question-answer (QA) pairs to enable scalable, crowd-sourced annotation of discourse relations. By framing discourse relations as QA pairs with question prefixes indicating relation types, the authors collect a large-scale, open-source dataset of over 16,600 QA pairs from ~9,000 sentences and establish baseline models for predicting discourse-aware questions and answers, achieving 83.69% F1 on the test set using a BERT-based pipeline with pointer-generator question generation and fine-tuned SQuAD answer prediction.

ABSTRACT

Discourse relations describe how two propositions relate to one another, and identifying them automatically is an integral part of natural language understanding. However, annotating discourse relations typically requires expert annotators. Recently, different semantic aspects of a sentence have been represented and crowd-sourced via question-and-answer (QA) pairs. This paper proposes a novel representation of discourse relations as QA pairs, which in turn allows us to crowd-source wide-coverage data annotated with discourse relations, via an intuitively appealing interface for composing such questions and answers. Based on our proposed representation, we collect a novel and wide-coverage QADiscourse dataset, and present baseline algorithms for predicting QADiscourse relations.

Motivation & Objective

• To address the high cost and low scalability of expert-annotated discourse relation datasets by enabling crowd-sourced annotation using intuitive QA pairs.
• To develop a representation of discourse relations as QA pairs where the question prefix encodes the relation sense, enabling non-experts to annotate complex relations.
• To collect a wide-coverage, open-source dataset of discourse-aware QA pairs from natural language sentences.
• To establish baseline models for predicting both the question (with relation sense) and answer (discourse unit) from input sentences.

Proposed method

• Represent discourse relations as QA pairs, where the question prefix encodes the relation sense (e.g., 'Why?' for cause), and the answer is the target discourse unit.
• Design a crowd-sourcing interface that guides non-experts to generate meaningful QA pairs by focusing on relation sense and discourse unit alignment.
• Use a BERT-encoding pointer-generator model to generate question bodies from sentence spans and relation prefixes, enabling copy and generation of words.
• Fine-tune a BERT model on SQuAD and a subset of the QADiscourse data to predict answer spans from gold questions.
• Apply a two-stage pipeline: first generate questions, then predict answers, with loss functions weighted for class imbalance.
• Evaluate using UQA (unlabeled QA alignment) and LQA (labeled accuracy) metrics, including prefix accuracy for relation sense prediction.

Experimental results

Research questions

• RQ1 Can discourse relations be effectively represented and crowd-sourced using a QA pair format where the question prefix encodes the relation sense?
• RQ2 Can non-expert annotators reliably produce high-quality discourse-aware QA pairs using this QA-based interface?
• RQ3 How effective are neural models in jointly predicting discourse-aware questions and answers from raw sentences?
• RQ4 To what extent does the proposed QA-based representation preserve the semantic and structural nuances of discourse relations compared to traditional annotation schemes?
• RQ5 What are the performance limits of baseline models in predicting both question prefixes and answer spans in the QADiscourse framework?

Key findings

• The QADiscourse dataset contains over 16,600 QA pairs collected from approximately 9,000 sentences, representing a wide-coverage, open-source resource for discourse relation annotation.
• The full pipeline baseline achieved 83.69% F1 on the test set using UQA evaluation, demonstrating the feasibility of end-to-end discourse QA prediction.
• Prefix accuracy for relation sense prediction was 49.94% on the test set, indicating significant room for improvement in sense classification.
• Answer prediction accuracy reached 72.3% on the test set when given gold questions, showing strong performance in span extraction.
• Qualitative analysis revealed that 44% of sampled predictions were judged correct, with models generalizing well to unseen verbs and forming questions from non-adjacent spans.
• The model produced 8 non-grammatical but semantically sensible QAs and 8 grammatical but nonsensical ones, highlighting challenges in fluency and meaning preservation.