[Paper Review] Enhancing Dialogue Generation via Multi-Level Contrastive Learning

Xin Li, Piji Li+3 more|arXiv (Cornell University)|Sep 19, 2020

Topic Modeling39 references4 citations

TL;DR

This paper proposes a multi-level contrastive learning framework that enhances dialogue generation by modeling fine-grained response quality at both token and instance levels. It introduces a Rank-aware Calibration (RC) network for token-level quality estimation via Monte-Carlo rollout and a Knowledge Inference (KI) component to promote informative word generation, resulting in more relevant and diverse responses than baseline models on a curated dialogue dataset.

ABSTRACT

Most of the existing works for dialogue generation are data-driven models trained directly on corpora crawled from websites. They mainly focus on improving the model architecture to produce better responses but pay little attention to considering the quality of the training data contrastively. In this paper, we propose a multi-level contrastive learning paradigm to model the fine-grained quality of the responses with respect to the query. A Rank-aware Calibration (RC) network is designed to construct the multi-level contrastive optimization objectives. Since these objectives are calculated based on the sentence level, which may erroneously encourage/suppress the generation of uninformative/informative words. To tackle this incidental issue, on one hand, we design an exquisite token-level strategy for estimating the instance loss more accurately. On the other hand, we build a Knowledge Inference (KI) component to capture the keyword knowledge from the reference during training and exploit such information to encourage the generation of informative words. We evaluate the proposed model on a carefully annotated dialogue dataset and the results suggest that our model can generate more relevant and diverse responses compared to the baseline models.

Motivation & Objective

• Address the limitation of existing dialogue models that treat all training samples equally, ignoring fine-grained response quality differences.
• Overcome the issue of uniform token weighting in response quality scoring, which may suppress informative words or encourage generic responses.
• Improve response relevance and diversity by explicitly modeling informative keywords from gold-standard responses during training.
• Develop a contrastive learning paradigm that integrates both instance-level and token-level quality supervision to guide sequence generation.

Proposed method

• Design a Rank-aware Calibration (RC) network that uses pointwise regression and pairwise ranking to estimate multi-scale response quality scores for query-response pairs.
• Implement a token-level quality estimation strategy using Monte-Carlo Rollout to conditionally sample responses and assign variable importance scores to individual tokens based on their contribution to quality.
• Introduce a Knowledge Inference (KI) component that captures and retains informative keywords from gold-standard responses by associating them with query and decoder hidden states.
• Incorporate summarized knowledge memories into each decoding step to encourage generation of salient, contextually relevant words.
• Combine RC and KI components within a contrastive learning framework to jointly optimize response relevance and informativeness.
• Use a multi-level contrastive objective that integrates both response-level and token-level quality supervision to refine model training.

Experimental results

Research questions

• RQ1 Can modeling fine-grained response quality at the token level improve the relevance and diversity of generated responses in dialogue systems?
• RQ2 How does token-level quality calibration via Monte-Carlo Rollout compare to standard instance-level weighting in preventing the suppression of informative words?
• RQ3 To what extent does explicitly learning and injecting keyword knowledge from reference responses enhance the informativeness of generated responses?
• RQ4 Does the integration of multi-level contrastive learning lead to consistent improvements over standard sequence-to-sequence models and existing quality-aware baselines?

Key findings

• The proposed model with both RC and KI components achieves the best performance across all evaluation metrics, outperforming standard S2S and baseline quality-aware models.
• Removing the RC component leads to a significant drop in relevance scores—by 6% on average, 6% on Extrema, and 7% on Greedy—demonstrating the importance of token-level calibration.
• The KI component effectively promotes the generation of meaningful keywords such as 'Fantasy Westward Journey' and 'Baidu', which are present in gold-standard responses but often missed by baselines.
• Visualizations confirm that the RC network assigns higher quality scores to semantically relevant and informative tokens (e.g., 'Sichuan' when discussing hometown, 'cloudy' in weather-related queries), validating its ability to capture context-dependent importance.
• The model reduces the generation of generic responses like 'oh, yes' or 'I also want to know', due to the fine-grained quality supervision that discourages low-quality, repetitive outputs.
• Ablation studies show that both RC and KI components are essential: removing either degrades performance, with RC having a more pronounced negative impact on relevance.