[Paper Review] Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents

Yi‐Chia Wang, Alexandros Papangelis+4 more|arXiv (Cornell University)|Dec 29, 2020

Topic Modeling72 references6 citations

TL;DR

This paper proposes a sequence-to-sequence deep learning model enhanced with a social language understanding component to inject politeness and positivity into task-oriented conversational agents while preserving task-relevant content. Evaluation shows the model successfully improves politeness in generated responses, correlating with higher user responsiveness and task completion, though positivity enhancement showed inconsistent results due to a mismatch between sentiment analysis and human perception of positivity.

ABSTRACT

Goal-oriented conversational agents are becoming prevalent in our daily lives. For these systems to engage users and achieve their goals, they need to exhibit appropriate social behavior as well as provide informative replies that guide users through tasks. The first component of the research in this paper applies statistical modeling techniques to understand conversations between users and human agents for customer service. Analyses show that social language used by human agents is associated with greater users' responsiveness and task completion. The second component of the research is the construction of a conversational agent model capable of injecting social language into an agent's responses while still preserving content. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.

Motivation & Objective

• To investigate whether social language in human-agent customer service conversations affects user responsiveness and task completion.
• To develop a deep learning model that generates task-oriented responses with controlled levels of politeness and positivity while preserving essential content.
• To evaluate the model’s effectiveness in maintaining content accuracy and enhancing social appropriateness through both automatic metrics and human judgment.
• To identify discrepancies between automated sentiment detection and human perception of positivity in dialogue systems.
• To provide a framework for integrating social language into task-oriented agents to improve user experience and engagement.

Proposed method

• The model uses a sequence-to-sequence neural network architecture with an added social language understanding module to control politeness and positivity levels in generated responses.
• Politeness and positivity are measured using pre-trained machine learning models from prior literature to analyze real human-human customer service conversations.
• The training data consists of real dialogues from a ride-sharing onboarding support system, annotated for social language use.
• The model is fine-tuned to generate responses with desired levels of politeness or positivity, using loss functions that balance content preservation and social language injection.
• Human evaluation via crowdsourcing assesses response quality, with raters judging politeness and positivity on a scale, while automatic metrics validate content preservation.
• The framework is evaluated using both automatic linguistic measures and human judgments to assess content fidelity and social language effectiveness.

Experimental results

Research questions

• RQ1 Does the use of social language by human agents in task-oriented conversations correlate with increased user responsiveness and task completion?
• RQ2 Can a deep learning model generate task-oriented responses that maintain content accuracy while effectively injecting politeness and positivity?
• RQ3 How do automated sentiment classifiers compare to human perception in measuring positivity in conversational responses?
• RQ4 What is the impact of politeness and positivity on user engagement in real-world task-oriented dialogues?
• RQ5 Can the proposed model improve real-world customer service outcomes by enhancing social language in agent responses?

Key findings

• Politeness in human agent responses was significantly correlated with higher driver responsiveness and increased completion of the first trip.
• Positivity showed a positive correlation with task completion but a negative correlation with responsiveness, suggesting a complex relationship with user behavior.
• The model successfully enhanced the politeness level of generated responses, with 87% of human raters judging enhanced responses as more polite than unenhanced ones.
• The model did not significantly improve perceived positivity, as only 13% of raters judged positivity-enhanced responses as more positive, indicating a mismatch between sentiment analysis and human perception.
• Inter-rater agreement was moderate for politeness but low for positivity (kappa = .2), suggesting ambiguity in defining and assessing positivity in dialogue contexts.
• The findings suggest that while sentiment-based models can detect politeness effectively, they may not align well with human understanding of positivity, warranting more targeted positivity modeling in future work.