[Paper Review] CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

Tao Yu, Rui Zhang+22 more|arXiv (Cornell University)|Sep 11, 2019

Topic Modeling59 references44 citations

TL;DR

CoSQL introduces a large-scale, cross-domain conversational text-to-SQL corpus collected via Wizard-of-Oz setup, with three tasks (SQL-grounded DST, NL response generation from SQL/results, and user dialogue act prediction) and baseline evaluations showing significant room for improvement.

ABSTRACT

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

Motivation & Objective

• Develop a cross-domain, general-purpose database querying dialogue corpus grounded in executable SQL representations.
• Enable evaluation of dialogue states grounded in SQL, NL response generation from SQL results, and user/dialogue-act prediction.
• Assess baseline models to reveal challenges in natural, clarifying, and verifiable SQL-based DB querying in a dialogue setting.
• Highlight cross-domain generalization to unseen databases and the necessity of system responses that explain SQL and results.

Proposed method

• Collect a large-scale, cross-domain WOZ dialogue dataset with 3,007 dialogues and 30k+ turns across 200 databases spanning 138 domains.
• Annotate each turn with executable SQL queries (10k labeled queries) and dialogue acts.
• Define three task benchmarks: SQL-grounded dialogue state tracking, response generation from executed SQL and results, and user dialogue act prediction.
• Benchmark strong baselines (e.g., CD-Seq2Seq and SyntaxSQL-con) for DST, and assess response generation with template-based, Seq2Seq, and pointer-generator approaches.
• Provide analysis on data diversity, cross-domain generalization, and complexity of SQL used in the dialogues.

Experimental results

Research questions

• RQ1 Can a system determine if a user question can be answered by SQL given cross-domain schemas?
• RQ2 How well can models ground user utterances into executable SQL queries in unseen databases?
• RQ3 Can a system generate natural-language responses that faithfully describe SQL queries and results for verification?
• RQ4 How accurately can user dialogue acts (e.g., ambiguous, infer_sql, cannot_understand) be predicted in this setup?

Key findings

• Baseline DST models achieve less than 16% question-level accuracy and less than 3% interaction-level accuracy on CoSQL, indicating a higher difficulty than SParC.
• Response-generation baselines show limited BLEU scores and low logic-correctness; pointer-generator often outperforms vanilla Seq2Seq, but all struggle to preserve SQL logic in NL outputs.
• About 40% of user questions are AMBIGUOUS and require system clarification; around 10% involve INFER_SQL, underscoring the need for effective clarification and inference in NLIDB systems.
• Human evaluation reveals that template-based NL descriptions can be more grammatically coherent but have limited coverage and lower logic-correctness than neural methods; neural methods yield higher BLEU but still struggle with logical fidelity.
• CoSQL spans 200 databases and 1020 tables in the training set, with long-tail language and diverse discourse, highlighting cross-domain generalization challenges.