[Paper Review] Emily: Developing An Emotion-affective Open-Domain Chatbot with Knowledge Graph-based Persona

Weixuan Wang, Xiaoling Cai+5 more|arXiv (Cornell University)|Sep 18, 2021

Topic Modeling28 references4 citations

TL;DR

This paper presents Emily, an emotion-affective open-domain chatbot that detects users' negative emotions and responds with empathetic, mood-enhancing replies by fine-tuning a dialogue model on data capturing desired emotion state transitions. It ensures personality consistency by using a knowledge graph-based question-answering (KGQA) system for persona-related queries, achieving 99.5% utterance classification accuracy and 97.78% hit@1 on persona QA.

ABSTRACT

In this paper, we describe approaches for developing Emily, an emotion-affective open-domain chatbot. Emily can perceive a user's negative emotion state and offer supports by positively converting the user's emotion states. This is done by finetuning a pretrained dialogue model upon data capturing dialogue contexts and desirable emotion states transition across turns. Emily can differentiate a general open-domain dialogue utterance with questions relating to personal information. By leveraging a question-answering approach based on knowledge graphs to handle personal information, Emily maintains personality consistency. We evaluate Emily against a few state-of-the-art open-domain chatbots and show the effects of the proposed approaches in emotion affecting and addressing personality inconsistency.

Motivation & Objective

• To develop an open-domain chatbot capable of perceiving and positively influencing users' emotional states during conversation.
• To address personality inconsistency in open-domain chatbots by decoupling persona-related queries from generative responses.
• To design a retrieval-based knowledge graph question-answering (KGQA) system that ensures accurate and consistent responses to personal information queries.
• To evaluate Emily’s effectiveness in emotion modulation and personality consistency against state-of-the-art open-domain chatbots.

Proposed method

• Fine-tuned a pretrained dialogue model (DialoGPT) on a dataset annotated with user emotion states (positive, neutral, negative) derived from Twitter and empathetic dialogue data.
• Used a RoBERTa-based utterance classifier to detect persona-related questions, achieving 99.5% accuracy in distinguishing personal from general queries.
• Implemented an emotion-affective module that generates responses to shift users’ emotion states from negative to positive, trained on dialogue contexts with desired emotion transitions.
• Built a knowledge graph (KG) embedding-based QA system (Embedded KGQA) to retrieve answers to persona and celebrity-related questions, using hit@1, hit@3, and hit@10 as evaluation metrics.
• Integrated a world celebrity knowledge graph to support responses on public figures, enhancing the chatbot’s factual grounding.
• Employed a multi-stage pipeline: utterance classification → emotion-affective response generation or KGQA retrieval, depending on query type.

Experimental results

Research questions

• RQ1 Can a fine-tuned dialogue model effectively generate responses that positively influence users’ emotional states in open-domain conversations?
• RQ2 How can personality consistency be preserved when handling personal information queries in open-domain chatbots?
• RQ3 To what extent does a knowledge graph-based QA system outperform generative models in answering persona-related questions?
• RQ4 Can a hybrid approach—combining generative response generation with retrieval-based QA—improve both emotional responsiveness and factual consistency?

Key findings

• The utterance classifier achieved 99.5% accuracy in distinguishing persona-related from general queries, enabling effective routing to the appropriate module.
• The emotion-affective module successfully generated responses that shifted users’ emotion states from negative to positive, as validated through emotion transition modeling.
• The Embedded KGQA system achieved a hit@1 score of 0.9778 on persona-related questions, indicating high accuracy in retrieving correct answers from the knowledge graph.
• The model demonstrated superior performance in personality consistency, significantly reducing discrepancies between persona information and generated responses.
• Emily outperformed several state-of-the-art open-domain chatbots in both emotion modulation and consistency, as confirmed by qualitative and quantitative evaluation.
• The integration of a knowledge graph for personal and celebrity information improved factual accuracy and reduced hallucination in responses.