[Paper Review] Reinforcement Learning Based Emotional Editing Constraint Conversation Generation

Jia Li, Xiao Sun+3 more|arXiv (Cornell University)|Apr 17, 2019

Topic Modeling15 references18 citations

TL;DR

This paper proposes a reinforcement learning-based dialogue generation model that integrates emotional editing constraints to produce more fluent, contextually relevant, and emotionally expressive replies. By dividing response generation into three stages and using an emotional editor to refine outputs based on pre-defined keywords and emotion templates, the model significantly improves emotional relevance and coherence, outperforming existing methods in fluency, topic consistency, and emotion accuracy.

ABSTRACT

In recent years, the generation of conversation content based on deep neural networks has attracted many researchers. However, traditional neural language models tend to generate general replies, lacking logical and emotional factors. This paper proposes a conversation content generation model that combines reinforcement learning with emotional editing constraints to generate more meaningful and customizable emotional replies. The model divides the replies into three clauses based on pre-generated keywords and uses the emotional editor to further optimize the final reply. The model combines multi-task learning with multiple indicator rewards to comprehensively optimize the quality of replies. Experiments shows that our model can not only improve the fluency of the replies, but also significantly enhance the logical relevance and emotional relevance of the replies.

Motivation & Objective

• To address the limitations of existing neural dialogue models that generate generic, emotionally flat replies.
• To improve emotional expressiveness in generated responses by incorporating explicit emotional editing constraints.
• To enhance response quality through multi-task learning that jointly optimizes fluency, topic relevance, and emotional accuracy.
• To develop a three-stage generation process that enables better control over emotional tone and content structure.

Proposed method

• The model generates replies in three iterative stages: initial response generation, keyword-based template selection, and emotional editing refinement.
• An emotional editor selects relevant sentence templates based on input topic and desired emotion, then optimizes the generated response using prior knowledge.
• Reinforcement learning is applied with multi-task rewards to jointly optimize fluency, topic coherence, and emotional relevance.
• The model uses a sequence-to-sequence framework enhanced with emotion category vectors and attention mechanisms for context modeling.
• A multi-task learning setup coordinates the optimization of three objectives: response fluency, topic consistency, and emotional accuracy.
• The emotional editing mechanism performs smoothing, rephrasing, and emotional intensification using pre-identified keywords and emotional templates.

Experimental results

Research questions

• RQ1 Can reinforcement learning with multi-task rewards improve both fluency and emotional relevance in dialogue generation?
• RQ2 How does incorporating an emotional editing stage affect the quality and expressiveness of generated responses?
• RQ3 To what extent can keyword-based template selection enhance emotional precision in open-domain dialogue systems?
• RQ4 Does the three-stage generation process reduce generic and repetitive responses compared to standard autoregressive models?
• RQ5 Can emotional editing constraints lead to more coherent and contextually appropriate replies without sacrificing fluency?

Key findings

• The proposed model achieves the lowest perplexity (61.4) and highest emotional accuracy (86.9%) among all compared models on the test set.
• The model outperforms SentiGAN and E-SCBA in both fluency and emotional accuracy, with a 10.2% improvement in emotional accuracy over the baseline Seq2Seq model.
• The inclusion of the emotional editor significantly improves performance, reducing perplexity by 3.6 points and increasing emotional accuracy by 9.4% compared to the model without editing.
• The model demonstrates superior word diversity across response positions, with slower fading of color intensity in word distribution visualizations, indicating better long-range coherence and reduced repetition.
• Case studies show that the model generates emotionally rich and contextually appropriate responses, such as 'I have never seen such a cute cake!' for happiness and 'Such bad weather!' for anger, unlike models that produce emotionally neutral replies.
• The model’s response quality is enhanced through the integration of prior knowledge (keywords and templates), resulting in more natural, fluent, and emotionally salient outputs.