[Paper Review] Dialogue Design and Management for Multi-Session Casual Conversation with Older Adults

Seyedeh Zahra Razavi, Lenhart K. Schubert+4 more|arXiv (Cornell University)|Jan 20, 2019

Social Robot Interaction and HRI18 references4 citations

TL;DR

This paper presents an automatic spoken dialogue manager for LISSA, a virtual agent designed to conduct multi-session casual conversations with older adults to improve their social communication skills. Using nonverbal feedback and topic progression, the system achieved high-quality, natural interactions rated as equal or better than wizard-mediated conversations, demonstrating strong usability and engagement in a pilot study with 8 participants.

ABSTRACT

We address the problem of designing a conversational avatar capable of a sequence of casual conversations with older adults. Users at risk of loneliness, social anxiety or a sense of ennui may benefit from practicing such conversations in private, at their convenience. We describe an automatic spoken dialogue manager for LISSA, an on-screen virtual agent that can keep older users involved in conversations over several sessions, each lasting 10-20 minutes. The idea behind LISSA is to improve users' communication skills by providing feedback on their non-verbal behavior at certain points in the course of the conversations. In this paper, we analyze the dialogues collected from the first session between LISSA and each of 8 participants. We examine the quality of the conversations by comparing the transcripts with those collected in a WOZ setting. LISSA's contributions to the conversations were judged by research assistants who rated the extent to which the contributions were "natural", "on track", "encouraging", "understanding", "relevant", and "polite". The results show that the automatic dialogue manager was able to handle conversation with the users smoothly and naturally.

Motivation & Objective

• To design a conversational virtual agent that supports older adults in practicing social communication skills over multiple sessions.
• To develop an automatic dialogue manager capable of sustaining natural, engaging, and contextually appropriate conversations with older users.
• To provide real-time feedback on nonverbal behaviors (e.g., eye contact, smiling) and emotional valence to help users improve communication effectiveness.
• To evaluate the system’s conversational quality and usability in a multi-session home-based study with older adults.
• To understand longitudinal user engagement, verbosity, self-disclosure, and sentiment across repeated interactions.

Proposed method

• The dialogue manager uses flexible schemas for planned and anticipated events, enabling structured yet adaptive conversation flow across 30 topics over 10 sessions.
• Hierarchical pattern transductions generate 'gist clause' interpretations and responses, allowing the system to derive contextually relevant replies.
• The system integrates real-time analysis of nonverbal cues—eye contact, head motion, smiling, and speech valence—to deliver feedback at two transition points and at the end of each session.
• Topics progress from low-emotional to higher-emotional disclosure, designed in collaboration with geriatric care experts to suit older adults’ cognitive and emotional needs.
• A wizard-of-Oz (WOZ) study was conducted to establish baseline dialogue quality, later compared to the fully automatic system’s performance.
• User sessions were conducted in a lab (first and last) and at home (intermediate sessions), with transcripts analyzed by research assistants for quality ratings.

Experimental results

Research questions

• RQ1 How does the quality of automatic dialogue compare to wizard-mediated dialogue in terms of naturalness, relevance, and engagement for older adults?
• RQ2 To what extent does the dialogue manager maintain conversational flow and user engagement across multiple sessions?
• RQ3 How do users’ verbosity, self-disclosure, and sentiment evolve over repeated interactions with the virtual agent?
• RQ4 What is the relationship between user nonverbal behavior and perceived conversational quality in a multi-session context?
• RQ5 How do users evaluate the system’s usability and perceived helpfulness in improving communication skills?

Key findings

• The automatic dialogue manager produced conversations rated as highly natural, on track, encouraging, understanding, relevant, and polite—matching or slightly exceeding the quality of wizard-mediated interactions.
• Research assistants rated the automatic system’s dialogue contributions significantly higher on average than those from the WOZ condition, indicating strong conversational quality.
• Users reported the system as easy to use and user-friendly, with many describing the interaction as similar to having a visitor at home.
• Participants appreciated the relatable topics, calm demeanor of the avatar, and the feedback on nonverbal behavior, which encouraged deeper, more reflective conversations.
• A small number of users requested longer conversations, suggesting potential for session duration extension in future iterations.
• Preliminary insights indicate that verbosity, self-disclosure, and sentiment may vary with topic and user personality, and are being further analyzed using ASR transcripts from at-home sessions.