[Paper Review] Deep Learning Based Chatbot Models

Richárd Csáky|arXiv (Cornell University)|Aug 23, 2019

Topic Modeling114 references39 citations

TL;DR

This paper surveys recent deep learning approaches to chatbots, analyzes encoder-decoder/Transformer-based methods, and proposes incorporating priors like mood and persona to improve open-domain dialogue generation.

ABSTRACT

A conversational agent (chatbot) is a piece of software that is able to communicate with humans using natural language. Modeling conversation is an important task in natural language processing and artificial intelligence. While chatbots can be used for various tasks, in general they have to understand users' utterances and provide responses that are relevant to the problem at hand. In my work, I conduct an in-depth survey of recent literature, examining over 70 publications related to chatbots published in the last 3 years. Then, I proceed to make the argument that the very nature of the general conversation domain demands approaches that are different from current state-of-of-the-art architectures. Based on several examples from the literature I show why current chatbot models fail to take into account enough priors when generating responses and how this affects the quality of the conversation. In the case of chatbots, these priors can be outside sources of information that the conversation is conditioned on like the persona or mood of the conversers. In addition to presenting the reasons behind this problem, I propose several ideas on how it could be remedied. The next section focuses on adapting the very recent Transformer model to the chatbot domain, which is currently state-of-the-art in neural machine translation. I first present experiments with the vanilla model, using conversations extracted from the Cornell Movie-Dialog Corpus. Secondly, I augment the model with some of my ideas regarding the issues of encoder-decoder architectures. More specifically, I feed additional features into the model like mood or persona together with the raw conversation data. Finally, I conduct a detailed analysis of how the vanilla model performs on conversational data by comparing it to previous chatbot models and how the additional features affect the quality of the generated responses.

Motivation & Objective

• Survey and synthesize over 70 publications on chatbots from the past 3 years.
• Argue that open-domain conversation requires priors beyond standard architectures.
• Experiment with Transformer-based chatbots on dialogue datasets to assess performance.
• Propose ideas to incorporate mood, persona, and other priors to improve response quality.

Proposed method

• Review historical and contemporary chatbot literature including encoder-decoder and Transformer models.
• Describe data preprocessing, word embeddings, and vocabulary handling in seq2seq frameworks.
• Experimentally train Transformer-based chatbots on Cornell Movie-Dialog Corpus and OpenSubtitles corpus.
• Augment encoder-decoder models with additional inputs such as mood and persona.
• Compare vanilla Transformer performance to prior chatbot models and analyze impact of priors.

Experimental results

Research questions

• RQ1 What are the limitations of current neural chatbot architectures for open-domain conversation?
• RQ2 Can Transformer-based architectures be effectively adapted to chatbot settings?
• RQ3 Do priors such as mood and persona improve the quality and relevance of generated responses?
• RQ4 How do context and conversation history influence Transformer-based chatbots?
• RQ5 What are effective strategies for incorporating knowledge bases and contextual information into chatbots?

Key findings

• Transformer-based chatbots can be trained on dialog datasets like the Cornell and OpenSubtitles corpora.
• Additional inputs (mood, persona) can be integrated to potentially improve response relevance and naturalness.
• Context and conversation history pose challenges for encoding and require hierarchical or memory-focused strategies.
• Evaluation of dialog models remains complex with traditional metrics (BLEU, perplexity) not always aligning with human judgments.
• The study discusses training setups and qualitative comparisons to prior chatbot models.
• Future work outlines methods to address loss function issues, temporal conditioning, and memory in dialog systems.