[Paper Review] Designing dialogue systems: A mean, grumpy, sarcastic chatbot in the browser

Suzana Ilić, Reiichiro Nakano+1 more|arXiv (Cornell University)|Sep 20, 2019

Topic Modeling4 citations

TL;DR

This paper presents a browser-deployed, LSTM-based seq2seq chatbot trained on a curated 3,000-sample dataset of sarcastic, humorous, and grumpy dialogue responses. Despite a small, low-vocabulary dataset, the model achieves high human-rated scores in sarcasm (71.3%), humor (73.8%), and personality (73.8%), demonstrating that end-to-end models can effectively learn and express complex emotional traits from minimal, high-quality data.

ABSTRACT

In this work we explore a deep learning-based dialogue system that generates sarcastic and humorous responses from a conversation design perspective. We trained a seq2seq model on a carefully curated dataset of 3000 question-answering pairs, the core of our mean, grumpy, sarcastic chatbot. We show that end-to-end systems learn patterns very quickly from small datasets and thus, are able to transfer simple linguistic structures representing abstract concepts to unseen settings. We also deploy our LSTM-based encoder-decoder model in the browser, where users can directly interact with the chatbot. Human raters evaluated linguistic quality, creativity and human-like traits, revealing the system's strengths, limitations and potential for future research.

Motivation & Objective

• To design a dialogue system with distinct personality traits—specifically sarcasm, humor, and grumpiness—through targeted data curation.
• To investigate whether end-to-end neural models can learn complex emotional and linguistic patterns from small, high-quality datasets rather than large, low-quality corpora.
• To enable interactive, real-time user evaluation of personality-driven chatbots via browser deployment using TensorFlow.js.
• To assess the perceived linguistic quality, creativity, and emotional expressiveness of generated responses through human evaluation.
• To explore the feasibility of generating emotionally expressive, non-task-oriented dialogue with minimal data and standard seq2seq architectures.

Proposed method

• Constructed a curated dataset of 3,000 question-answer pairs with humorous, sarcastic, and emotionally charged responses drawn from jokes, movie quotes, and social media.
• Employed an LSTM-based encoder-decoder architecture with greedy decoding to generate responses token-by-token from encoded context.
• Used word-level input with GloVe embeddings and experimented with attention mechanisms, though no significant qualitative improvement was observed.
• Deployed the trained model in the browser using TensorFlow.js to enable real-time, interactive user interaction and evaluation.
• Applied temperature sampling in the demo to introduce controlled randomness and explore variability in generated responses.
• Evaluated model outputs via human raters using a 1–10 scale across multiple dimensions: coherence, adequacy, creativity, lexical variation, sarcasm, humor, personality, emotion, and context awareness.

Experimental results

Research questions

• RQ1 Can a small, carefully curated dataset of 3,000 dialogue pairs effectively train a seq2seq model to generate sarcastic and humorous responses?
• RQ2 To what extent can end-to-end neural models learn and express complex emotional traits like sarcasm and personality without large-scale, noisy datasets?
• RQ3 How do human raters perceive the linguistic quality, creativity, and emotional expressiveness of responses from a personality-driven chatbot?
• RQ4 What is the impact of architectural choices—such as attention mechanisms and embedding strategies—on the quality of emotionally expressive generation in small-data settings?
• RQ5 Can browser-based deployment enable scalable, interactive evaluation of personality-rich dialogue systems by diverse users?

Key findings

• The chatbot achieved the highest average human evaluation scores in personality (73.8%) and humor (73.8%), indicating strong perceived expressiveness and engagement.
• Sarcasm was rated highly at 71.3%, demonstrating the model’s ability to generate contextually appropriate, ironic responses despite limited training data.
• Creativity received an average score of 68.8%, suggesting that even with a small vocabulary (~1,400 tokens), the model produced novel and surprising responses.
• Lexical variation scored relatively low at 56.3%, indicating repetitive phrasing due to model memorization and greedy decoding, despite high coherence.
• 51.4% of responses were rated as 'match' (accurate and relevant), 24.8% as 'ambiguous', and 23.8% as 'nonsensical', highlighting challenges in response consistency.
• Emotion perception was rated lowest at 54.4%, suggesting that while sarcasm and humor were well-expressed, broader emotional nuance remains a challenge for the current setup.