[Paper Review] FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial Expressions

Neeraj Cherakara, Finny S. Varghese+10 more|arXiv (Cornell University)|Aug 29, 2023

Social Robot Interaction and HRIPsychology3 citations

TL;DR

FurChat is an embodied conversational agent built on GPT-3.5 that integrates open- and closed-domain dialogue with context-aware facial expressions on a Furhat robot, enabling natural, engaging interactions as a receptionist at the National Robotarium. The system uses prompt engineering to align LLM-generated responses with emotional expressions and dialogue flow, demonstrating improved engagement through multimodal output.

ABSTRACT

We demonstrate an embodied conversational agent that can function as a receptionist and generate a mixture of open and closed-domain dialogue along with facial expressions, by using a large language model (LLM) to develop an engaging conversation. We deployed the system onto a Furhat robot, which is highly expressive and capable of using both verbal and nonverbal cues during interaction. The system was designed specifically for the National Robotarium to interact with visitors through natural conversations, providing them with information about the facilities, research, news, upcoming events, etc. The system utilises the state-of-the-art GPT-3.5 model to generate such information along with domain-general conversations and facial expressions based on prompt engineering.

Motivation & Objective

• To develop an embodied conversational agent that supports both open- and closed-domain dialogue in a natural, human-like manner.
• To integrate expressive facial gestures with LLM-generated responses to enhance emotional engagement and social presence.
• To deploy the system on a Furhat robot for real-world interaction in a public research facility setting.
• To address hallucination in LLM responses through a curated database and prompt engineering.
• To demonstrate the feasibility of using LLMs for multimodal, socially intelligent robot interactions beyond rule-based systems.

Proposed method

• The system uses GPT-3.5 as the core natural language generation engine, prompted with context, personality, dialogue history, and response format including emoticon cues.
• A custom database of verified facts (e.g., about the National Robotarium) is used to ground closed-domain responses and reduce hallucination.
• The dialogue manager routes user inputs to the LLM after intent classification via FurhatOS’s NLU, then maps generated responses to pre-defined facial gestures.
• Facial expressions are triggered via gesture parsing using conditional logic based on sentiment and emotion detection in the LLM output.
• Text-to-speech is handled by Amazon Polly, integrated via FurhatOS, to produce natural-sounding speech synchronized with facial expressions.
• The system employs prompt engineering to embed personality, emotional tone, and expression rules directly into the LLM prompt for consistent multimodal output.

Figure 1: A user interacting with the FurChat System.

Experimental results

Research questions

• RQ1 Can an LLM-powered conversational agent effectively blend open-domain and closed-domain dialogue in a real-world receptionist setting?
• RQ2 How can facial expressions be systematically generated and synchronized with LLM-generated speech to enhance perceived social presence?
• RQ3 To what extent can prompt engineering and a curated knowledge base reduce hallucination in LLM-driven embodied agents?
• RQ4 Can a single LLM-based system maintain coherence and engagement across diverse interaction types, from factual Q&A to casual conversation?
• RQ5 How does the integration of multimodal cues (speech, facial expression) affect user perception of naturalness and trust in a social robot?

Key findings

• The FurChat system successfully supports a seamless mix of open-domain and closed-domain dialogue, enabling natural, flowing conversations with visitors.
• Facial expressions were effectively synchronized with speech and emotional context, enhancing perceived expressiveness and social engagement.
• The use of a curated database significantly reduced hallucinated content in closed-domain responses, improving factual reliability.
• Prompt engineering enabled consistent personality expression and emotion alignment, resulting in coherent and contextually appropriate responses.
• The system demonstrated robust performance in real-world deployment at the National Robotarium, handling diverse visitor queries with minimal failure.
• The integration of GPT-3.5 with the Furhat SDK enabled a scalable, extensible framework for multimodal, embodied conversation beyond rule-based systems.

Figure 2: System Architecture of the current FurChat system.