[Paper Review] DialBERT: A Hierarchical Pre-Trained Model for Conversation Disentanglement

Tianda Li, Jia-Chen Gu+5 more|arXiv (Cornell University)|Apr 8, 2020

Topic Modeling28 references19 citations

TL;DR

DialBERT is a hierarchical pre-trained model that improves conversation disentanglement by integrating local utterance-level matching via BERT and global context-level modeling via a BiLSTM. With only a 3% parameter increase, it achieves a 12% F1-score improvement over BERT and sets a new state-of-the-art on the IBM Kummerfeld (2019) dataset, outperforming prior models across all metrics by a substantial margin.

ABSTRACT

Disentanglement is a problem in which multiple conversations occur in the same channel simultaneously, and the listener should decide which utterance is part of the conversation he will respond to. We propose a new model, named Dialogue BERT (DialBERT), which integrates local and global semantics in a single stream of messages to disentangle the conversations that mixed together. We employ BERT to capture the matching information in each utterance pair at the utterance-level, and use a BiLSTM to aggregate and incorporate the context-level information. With only a 3% increase in parameters, a 12% improvement has been attained in comparison to BERT, based on the F1-Score. The model achieves a state-of-the-art result on the a new dataset proposed by IBM and surpasses previous work by a substantial margin.

Motivation & Objective

• To address the challenge of disentangling mixed conversations in single-channel messaging where multiple dialogues occur simultaneously.
• To improve disentanglement performance by integrating both utterance-level and context-level semantic representations.
• To develop a domain-adapted pre-trained model effective for the Ubuntu forum domain, leveraging dialogue-specific fine-tuning.
• To surpass existing models on a new large-scale disentanglement benchmark, demonstrating superior generalization and robustness.

Proposed method

• The model uses BERT to compute local semantic matching between candidate message pairs at the utterance level.
• A BiLSTM is applied to encode global context by aggregating information across preceding messages in the conversation stream.
• The model is post-trained on Ubuntu forum data using masked language modeling (MLM) and next sentence prediction (NSP) objectives to adapt to dialogue-specific semantics.
• Similarity scores between target and context messages are computed to identify the correct preceding message, with prediction based on the highest score.
• The architecture processes messages in a single stream, enabling end-to-end learning of both local and global dependencies.
• Ensemble strategies (probability averaging, model averaging) are applied post-training to further improve performance.

Experimental results

Research questions

• RQ1 Can a hierarchical pre-trained model that combines local utterance matching and global context modeling outperform standard BERT in conversation disentanglement?
• RQ2 How effective is domain-specific post-training on Ubuntu forum data for improving disentanglement performance?
• RQ3 To what extent do linguistic features contribute to performance when a strong pre-trained model like DialBERT is already used?
• RQ4 How critical is context modeling via BiLSTM for disentanglement, and what happens when it is removed?
• RQ5 Can the model generalize well on a new, large-scale disentanglement benchmark, surpassing previous state-of-the-art approaches?

Key findings

• DialBERT achieves a 12% relative improvement in F1-score over standard BERT, with gains across all six evaluation metrics.
• The model sets a new state-of-the-art on the IBM Kummerfeld (2019) disentanglement dataset, outperforming all prior models.
• Post-training on Ubuntu forum data significantly improves performance, with a 3% parameter increase yielding substantial gains.
• Removing the BiLSTM layer causes a sharp drop in performance (F1 from 43.9% to 32.5%), confirming its critical role in context modeling.
• Linguistic features provide minimal improvement when used with DialBERT, indicating the model already captures most relevant discourse cues.
• Ensemble methods like model averaging and probability averaging further boost performance, with the best ensemble achieving 45.3% F1, though the task remains challenging.