[Paper Review] ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)

Mohammad Aliannejadi, Julia Kiseleva+3 more|arXiv (Cornell University)|Sep 23, 2020

Topic Modeling9 references32 citations

TL;DR

The paper describes the ClariQ challenge for generating and evaluating clarifying questions in open-domain dialogue, with Stage 1 static data and Stage 2 human-in-the-loop evaluation to study when to ask clarifications and how to generate them.

ABSTRACT

This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ). The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020. The main aim of the conversational systems is to return an appropriate answer in response to the user requests. However, some user requests might be ambiguous. In IR settings such a situation is handled mainly thought the diversification of the search result page. It is however much more challenging in dialogue settings with limited bandwidth. Therefore, in this challenge, we provide a common evaluation framework to evaluate mixed-initiative conversations. Participants are asked to rank clarifying questions in an information-seeking conversations. The challenge is organized in two stages where in Stage 1 we evaluate the submissions in an offline setting and single-turn conversations. Top participants of Stage 1 get the chance to have their model tested by human annotators.

Motivation & Objective

• Motivate the need for clarifying questions in open-domain dialogue to handle ambiguous user requests.
• Propose a two-stage ClariQ challenge to study when to ask for clarification and how to generate clarifying questions.
• Provide datasets, evaluation metrics, and a human-in-the-loop protocol to benchmark approaches.

Proposed method

• Introduce Stage 1 with static datasets comprising initial user request, clarifying question options, and user answers; train/validate/test splits are provided.
• Define two tasks: (a) estimate a 1–4 score indicating the necessity of asking a clarifying question; (b) select the most suitable clarifying question from a bank.
• Use retrieval-style evaluation metrics (MRR, P@k, nDCG) to assess how clarifications affect downstream document retrieval.
• Stage 2 engages real users (human-in-the-loop) to evaluate conversational systems that must decide when to clarify and how to respond, favoring generative models for dialogue management.
• Provide a public repository with data and evaluation scripts.

Experimental results

Research questions

• RQ1 RQ1: When to ask clarifying questions during dialogues?
• RQ2 RQ2: How to generate the clarifying questions?

Key findings

• Stage 1 provides datasets with user requests, clarifying questions, and answers, plus a 70/30 train/validation split and a test setup with a question bank.
• Evaluation shows performance via retrieval metrics (MRR, P@1/3/5/10/20, nDCG@1/3/5/20) and Recall@10/20/30 for question relevance.
• Stage 2 highlights a shift to generative modeling and real-user evaluation to balance asking clarifications with providing relevant answers.
• Human-in-the-loop evaluation collects conversation history, system response, and user ratings for relevance and naturalness.
• The repository at GitHub hosts data and automatic evaluation scripts.