[Paper Review] Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations

Sam Coope, Tyler Farghly+3 more|arXiv (Cornell University)|Jan 1, 2020

Topic Modeling36 references4 citations

TL;DR

Span-ConveRT proposes a lightweight, turn-based span extraction model for dialog slot-filling that leverages pretrained conversational representations from ConveRT, achieving consistent gains in few-shot learning scenarios. It outperforms both BERT-based span extractors and models fine-tuned from scratch, especially on low-resource data, and introduces the RESTAURANTS-8K dataset for benchmarking.

ABSTRACT

We introduce Span-ConveRT, a light-weight model for dialog slot-filling which frames the task as a turn-based span extraction task. This formulation allows for a simple integration of conversational knowledge coded in large pretrained conversational models such as ConveRT (Henderson et al., 2019). We show that leveraging such knowledge in Span-ConveRT is especially useful for few-shot learning scenarios: we report consistent gains over 1) a span extractor that trains representations from scratch in the target domain, and 2) a BERT-based span extractor. In order to inspire more work on span extraction for the slot-filling task, we also release RESTAURANTS-8K, a new challenging data set of 8,198 utterances, compiled from actual conversations in the restaurant booking domain.

Motivation & Objective

• Address the challenge of low-resource, few-shot slot-filling in task-oriented dialog systems.
• Develop a flexible, intent-agnostic span extraction framework independent of other dialog components.
• Leverage conversational knowledge from large-scale pretrained models like ConveRT to improve few-shot performance.
• Introduce a new, realistic benchmark dataset—RESTAURANTS-8K—collected from real-world restaurant booking conversations.
• Enable robust, data-efficient slot-filling by transferring conversational representations instead of training from scratch.

Proposed method

• Frame slot-filling as a turn-based span extraction task, where each slot is predicted as a span or absent.
• Use ConveRT, a large-scale conversational model pre-trained on Reddit data, to provide contextual representations.
• Integrate slot request status as an input feature to model turn-level context without relying on intent detection.
• Apply a CNN-CRF architecture on top of ConveRT embeddings for end-to-end span prediction.
• Transfer subword representations from ConveRT to the target domain with minimal fine-tuning.
• Use character-level span indexing for all slot values, enabling flexible and accurate extraction of natural language expressions.

Experimental results

Research questions

• RQ1 Can leveraging pretrained conversational representations from ConveRT improve few-shot span extraction in dialog slot-filling?
• RQ2 How does Span-ConveRT compare to BERT-based span extractors and models trained from scratch in low-data regimes?
• RQ3 Does using a conversational pretraining objective yield better performance than standard language modeling pretraining for dialog tasks?
• RQ4 How effective is the RESTAURANTS-8K dataset as a benchmark for evaluating span extraction in realistic dialog settings?
• RQ5 What types of errors are most common in span extraction, and how do they differ between models in low- and high-data settings?

Key findings

• Span-ConveRT achieves consistent improvements over a BERT-based span extractor across all few-shot settings on both RESTAURANTS-8K and DSTC8 datasets.
• On the RESTAURANTS-8K dataset, Span-ConveRT reaches an F1 score of 0.95 with 100% training data and maintains strong performance down to 1/16th of the data (F1 = 0.86), outperforming the baseline.
• With only 64 training examples (0.75% of the full set), Span-ConveRT achieves an F1 of 0.86, significantly outperforming the BERT-based model and the model trained from scratch.
• The model trained from scratch on the target domain performs poorly in low-data regimes, with F1 dropping to 0.58 at 1/16th of the data, highlighting the need for pretraining.
• Conversational pretraining with ConveRT yields stronger performance than standard BERT pretraining, even when BERT is fine-tuned on full data, indicating that conversational pretraining is better aligned with dialog tasks.
• Error analysis shows that Span-ConveRT reduces misclassification of out-of-vocabulary words as names, particularly improving performance on the 'first name' slot in low-data settings.