[Paper Review] Interpretation of Natural Language Rules in Conversational Machine Reading

Marzieh Saeidi, Max Bartolo+6 more|arXiv (Cornell University)|Aug 28, 2018

Topic Modeling30 references4 citations

TL;DR

This paper introduces Conversational Machine Reading (CMR), a task requiring models to interpret natural language rules and apply background knowledge to answer 'Can I...?' or 'Do I have to...?' questions through interactive dialogues. Using a novel crowd-sourcing protocol with virtual users, the authors collect a 32k-instance dataset and show that while models perform well without background knowledge, significant improvement is needed when such knowledge is required, highlighting the challenge of rule interpretation in real-world QA.

ABSTRACT

Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems require the reading of text not because it contains the literal answer, but because it contains a recipe to derive an answer together with the reader's background knowledge. One example is the task of interpreting regulations to answer "Can I...?" or "Do I have to...?" questions such as "I am working in Canada. Do I have to carry on paying UK National Insurance?" after reading a UK government website about this topic. This task requires both the interpretation of rules and the application of background knowledge. It is further complicated due to the fact that, in practice, most questions are underspecified, and a human assistant will regularly have to ask clarification questions such as "How long have you been working abroad?" when the answer cannot be directly derived from the question and text. In this paper, we formalise this task and develop a crowd-sourcing strategy to collect 32k task instances based on real-world rules and crowd-generated questions and scenarios. We analyse the challenges of this task and assess its difficulty by evaluating the performance of rule-based and machine-learning baselines. We observe promising results when no background knowledge is necessary, and substantial room for improvement whenever background knowledge is needed.

Motivation & Objective

• To formalize a new task—Conversational Machine Reading (CMR)—where systems interpret natural language rules and apply background knowledge to answer complex 'Can I...?' or 'Do I have to...?' questions.
• To address the challenge of underspecified questions by modeling dialogues that include clarification questions and answers.
• To develop a scalable, controlled crowd-sourcing protocol using virtual users to generate high-quality, diverse dialogues around real regulatory rules.
• To create a large-scale, real-world dataset of 32k task instances based on actual UK government regulations and crowd-generated questions and scenarios.
• To evaluate baseline models and demonstrate that performance drops significantly when background knowledge is required, indicating a key challenge for future research.

Proposed method

• Design a dialog-based annotation interface where annotators collaboratively build conversations with virtual users who respond to follow-up questions using pre-defined answers.
• Control the ratio of 'Yes' and 'No' answers and the content of dialog history by managing virtual user responses, ensuring balanced and diverse data collection.
• Collect 32k task instances from real regulatory texts (e.g., UK National Insurance rules), with each instance including an initial question, a scenario, a rule snippet, dialog history, and a final answer or follow-up question.
• Use a sequence-to-sequence (seq2seq) neural model with and without copy mechanisms to generate answers or clarification questions, encoding context, history, and question inputs.
• Train and evaluate models on the CMR task using metrics including macro accuracy, BLEU-1, BLEU-4, and a custom metric (CM) to assess answer quality and fluency.
• Conduct a user study comparing CMR-assisted QA to manual rule reading, measuring speed and accuracy of decision-making.

Experimental results

Research questions

• RQ1 Can models effectively interpret natural language rules in regulatory texts to answer complex 'Can I...?' or 'Do I have to...?' questions in a conversational setting?
• RQ2 How does the inclusion of background knowledge affect model performance in rule-based question answering?
• RQ3 To what extent can a controlled, virtual-user-based annotation protocol produce high-quality, diverse, and balanced dialog data for CMR?
• RQ4 How do neural sequence-to-sequence models with and without copy mechanisms perform on the CMR task, especially in generating clarification questions?
• RQ5 Does using a conversational agent significantly improve user speed and accuracy in answering regulatory questions compared to manual rule reading?

Key findings

• The proposed CMR task is significantly more challenging when background knowledge is required, as evidenced by a drop in model performance from 61.9% macro accuracy (no background knowledge) to 44.8% (with background knowledge).
• The user study shows that participants using the conversational agent reached correct conclusions over twice as fast (2x faster) and with 93% accuracy, compared to 68% accuracy when reading rules manually.
• The crowd-sourcing protocol achieved substantial agreement (Fleiss’ Kappa = 0.71) among three annotators, indicating reliable data collection with controlled virtual users.
• Models using copy mechanisms outperform those without, suggesting that copying key terms from the rule text improves answer generation and fluency.
• The best-performing model (CM) achieved a macro accuracy of 0.619 and BLEU-1 of 0.689 when no background knowledge was needed, indicating promising but incomplete performance.
• The results highlight a critical gap in current models: substantial room for improvement remains when reasoning over rules and integrating external knowledge, pointing to the need for better entailment and question generation mechanisms.