[Paper Review] Data Distillation for Controlling Specificity in Dialogue Generation

Jiwei Li, Will Monroe+1 more|arXiv (Cornell University)|Feb 22, 2017

Topic Modeling28 references18 citations

TL;DR

This paper proposes a data distillation approach to train multiple dialogue generation models with varying specificity levels from a single dataset, using iterative removal of responses closest to the model's most common outputs. A reinforcement learning agent then selects the most context-appropriate model at test time, resulting in higher-quality, more context-sensitive responses than baseline models.

ABSTRACT

People speak at different levels of specificity in different situations. Depending on their knowledge, interlocutors, mood, etc.} A conversational agent should have this ability and know when to be specific and when to be general. We propose an approach that gives a neural network--based conversational agent this ability. Our approach involves alternating between \emph{data distillation} and model training : removing training examples that are closest to the responses most commonly produced by the model trained from the last round and then retrain the model on the remaining dataset. Dialogue generation models trained with different degrees of data distillation manifest different levels of specificity. We then train a reinforcement learning system for selecting among this pool of generation models, to choose the best level of specificity for a given input. Compared to the original generative model trained without distillation, the proposed system is capable of generating more interesting and higher-quality responses, in addition to appropriately adjusting specificity depending on the context. Our research constitutes a specific case of a broader approach involving training multiple subsystems from a single dataset distinguished by differences in a specific property one wishes to model. We show that from such a set of subsystems, one can use reinforcement learning to build a system that tailors its output to different input contexts at test time.

Motivation & Objective

• To enable dialogue agents to vary response specificity based on context, mimicking human conversational behavior.
• To address the challenge of training models with controlled specificity when no explicit supervision for specificity exists.
• To develop a method that generates a spectrum of specificity levels from a single dataset without requiring manual labeling of specificity.
• To use reinforcement learning to dynamically select the optimal specificity level at test time based on input context.

Proposed method

• Iteratively train a sequence-to-sequence model on a dataset, then remove training examples whose outputs are semantically closest to the model’s most frequent responses.
• Repeat the training and distillation process across multiple rounds, with each subsequent model becoming more specific as generic responses are removed.
• Construct a pool of generation models, each trained on progressively more specific subsets of the original data, forming a spectrum of specificity.
• Train a reinforcement learning agent to select the best model from the pool for each input based on context, optimizing for response quality and specificity.
• Use human evaluation and adversarial metrics to compare the RL-selected model against baseline models and individual distillation models.

Experimental results

Research questions

• RQ1 Can iterative data distillation produce a spectrum of dialogue generation models with increasing specificity from a single dataset?
• RQ2 Can a reinforcement learning agent effectively select the most context-appropriate specificity level at inference time?
• RQ3 Does the RL-selected model generate higher-quality and more diverse responses than models trained without distillation or with fixed specificity?
• RQ4 How does the choice of specificity model correlate with input characteristics such as vagueness or unknown words?

Key findings

• Models trained in early rounds of data distillation (e.g., Iter2) produced higher-quality responses than the full-dataset baseline (Iter1), as confirmed by human evaluation.
• The reinforcement learning model achieved a 60% win rate in human evaluations, outperforming both Iter1 and Iter2 individually.
• The RL model selected the Iter1 model (least specific) for 16% of inputs, primarily when inputs were vague or ambiguous.
• The RL model preferred more specific models (e.g., Iter3, Iter4) when inputs were clear and contained more unknown words, indicating context-aware specificity selection.
• The RL model achieved 8.8% adversarial success and 94.4% machine-vs-random accuracy, outperforming both the baseline Iter1 (5.8% success) and random model selection.