[Paper Review] Multi-Task Self-Supervised Learning for Disfluency Detection

Shaolei Wang, Wanxiang Che+4 more|arXiv (Cornell University)|Aug 15, 2019

Speech and dialogue systems4 citations

TL;DR

This paper proposes a multi-task self-supervised learning framework for disfluency detection that leverages automatically generated pseudo-training data through two self-supervised tasks: tagging noisy words inserted into fluent sentences and classifying original from corrupted sentences. The method achieves state-of-the-art performance, reducing error by 21% on the Switchboard test set when trained on the full dataset and outperforming BERT even with 1% of the human-annotated data.

ABSTRACT

Most existing approaches to disfluency detection heavily rely on human-annotated data, which is expensive to obtain in practice. To tackle the training data bottleneck, we investigate methods for combining multiple self-supervised tasks-i.e., supervised tasks where data can be collected without manual labeling. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled news data, and propose two self-supervised pre-training tasks: (i) tagging task to detect the added noisy words. (ii) sentence classification to distinguish original sentences from grammatically-incorrect sentences. We then combine these two tasks to jointly train a network. The pre-trained network is then fine-tuned using human-annotated disfluency detection training data. Experimental results on the commonly used English Switchboard test set show that our approach can achieve competitive performance compared to the previous systems (trained using the full dataset) by using less than 1% (1000 sentences) of the training data. Our method trained on the full dataset significantly outperforms previous methods, reducing the error by 21% on English Switchboard.

Motivation & Objective

• To address the high cost and scarcity of human-annotated disfluency detection data.
• To investigate whether self-supervised pre-training can reduce reliance on expensive human annotations.
• To design self-supervised tasks that capture sentence-level grammatical structure and disfluency patterns.
• To improve disfluency detection performance using only a small fraction of the standard training data.
• To demonstrate the effectiveness of multi-task self-supervised learning in low-resource settings for NLP.

Proposed method

• Constructs large-scale pseudo-training data by randomly adding or deleting words from unlabeled news sentences.
• Proposes a tagging task to identify and remove inserted noisy words in disfluent sentences.
• Introduces a sentence classification task to distinguish original fluent sentences from grammatically incorrect, corrupted versions.
• Jointly trains a neural network on both self-supervised tasks to learn robust sentence representations.
• Fine-tunes the pre-trained model on a small amount of human-annotated disfluency detection data.
• Combines the pre-trained model with BERT by concatenating hidden representations for further performance gain.

Experimental results

Research questions

• RQ1 Can self-supervised pre-training with automatically generated data improve disfluency detection performance?
• RQ2 Do multi-task self-supervised objectives that emphasize grammatical correctness and fluency recovery enhance model generalization?
• RQ3 How effective is the proposed method when fine-tuned on only 1% of the standard human-annotated training data?
• RQ4 Can the proposed self-supervised model outperform or complement BERT in low-data regimes?
• RQ5 Does combining the proposed model with BERT further improve performance on disfluency detection?

Key findings

• The proposed method reduces error by 21% on the English Switchboard test set when trained on the full dataset, significantly outperforming previous methods.
• With only 1,000 human-annotated sentences (less than 1% of the full dataset), the method achieves a F1 score of 86.1%, surpassing BERT’s 82.4% on the same low-data setting.
• The model achieves 90.2% F1 on the full dataset, matching the best reported results and outperforming the previous state-of-the-art.
• The self-supervised tagging and classification tasks show strong performance, reaching 80% F1 and 97% accuracy respectively, confirming their learnability and usefulness.
• Combining the proposed model with BERT yields a further performance boost, achieving 91.4% F1 on the full dataset and 87.8% on 1,000 sentences, indicating complementary representations.
• The model performs particularly well on non-repetition disfluencies (70.8% F1), outperforming baselines, suggesting it captures complex structural patterns better than rule-based or random-initial models.