[Paper Review] Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models

Sheng-Chieh Lin, Jheng-Hong Yang+4 more|arXiv (Cornell University)|Apr 4, 2020

Topic ModelingComputer Science22 references36 citations

TL;DR

The paper empirically studies conversational question reformulation (CQR) using sequence-to-sequence models and pretrained language models, showing that T5 achieves state-of-the-art BLEU on CANARD and CAsT, approaching human performance on CANARD.

ABSTRACT

This paper presents an empirical study of conversational question reformulation (CQR) with sequence-to-sequence architectures and pretrained language models (PLMs). We leverage PLMs to address the strong token-to-token independence assumption made in the common objective, maximum likelihood estimation, for the CQR task. In CQR benchmarks of task-oriented dialogue systems, we evaluate fine-tuned PLMs on the recently-introduced CANARD dataset as an in-domain task and validate the models using data from the TREC 2019 CAsT Track as an out-domain task. Examining a variety of architectures with different numbers of parameters, we demonstrate that the recent text-to-text transfer transformer (T5) achieves the best results both on CANARD and CAsT with fewer parameters, compared to similar transformer architectures.

Motivation & Objective

• Motivate the need to rewrite questions in-context to resolve coreference and ellipsis in conversational QA and task-oriented dialogue.
• Investigate whether pretrained language models can relax the token-to-token independence assumption in MLE for CQR.
• Evaluate different PLMs and S2S architectures on CANARD (in-domain) and CAsT (out-of-domain) to identify strong performers.

Proposed method

• Formulate CQR as a sequence-to-sequence task that takes the current question plus dialogue history as input and outputs a rewritten question.
• Fine-tune a range of models (LSTM-based S2S, GPT-2, BERT, UniLM, and T5) on CANARD with concatenated history and current query as context.
• Use BLEU as the evaluation metric to compare outputs against human-rewritten references on CANARD and CAsT.
• Explore decoding strategies including greedy search, beam search (beam width up to 10), and top-k sampling to assess decoding sensitivity.
• Report model parameter counts and training setups for reproducibility.

Experimental results

Research questions

• RQ1 Can pretrained language models improve conversational question reformulation beyond a pure MLE baseline in CQR?
• RQ2 Which model architectures (and parameter scales) best leverage context/history to rewrite questions in CANARD and CAsT?
• RQ3 How does decoding strategy influence CQR quality as measured by BLEU on CANARD and CAsT?

Key findings

• T5-base achieves the best BLEU among tested models on CANARD (59.13) and CAsT (58.08) with Greedy decoding.
• T5-base with beam search (width=10) reaches BLEU of 76.22 on CAsT and is on par with human performance on CANARD (BLEU around 59.92 for humans, 58–60 range for models).
• On CANARD, humans outperform models, with Human at 59.92 BLEU and Raw at 33.84; various PLMs improve over Raw, e.g., UniLM-large at 57.39 (CANARD Dev) and 55.92 (CAsT Dev).
• GPT-2-medium and BERT-large also improve over LSTM, with GPT-2-medium at 52.63 (CANARD Dev) and 50.07 (CANARD Test); BERT-large at 55.34 (CANARD Dev) and 54.34 (CANARD Test).
• UniLM-large and T5-base consistently outperform other architectures across datasets, illustrating the benefit of text-to-text transfer models for CQR.