[Paper Review] Affective Faces for Goal-Driven Dyadic Communication

Scott Geng, Revant Teotia+3 more|arXiv (Cornell University)|Jan 26, 2023

Subtitles and Audiovisual MediaArts and Humanities3 citations

TL;DR

This paper proposes a framework that uses large language models (LLMs) and vision-language models (VLMs) to generate socially appropriate facial expressions for a listener in dyadic conversations, conditioned on the speaker's speech and the listener's goals or personality. It achieves significant improvements in social appropriateness over baselines, as validated through human evaluation and a new dataset of unscripted, diverse conversations.

ABSTRACT

We introduce a video framework for modeling the association between verbal and non-verbal communication during dyadic conversation. Given the input speech of a speaker, our approach retrieves a video of a listener, who has facial expressions that would be socially appropriate given the context. Our approach further allows the listener to be conditioned on their own goals, personalities, or backgrounds. Our approach models conversations through a composition of large language models and vision-language models, creating internal representations that are interpretable and controllable. To study multimodal communication, we propose a new video dataset of unscripted conversations covering diverse topics and demographics. Experiments and visualizations show our approach is able to output listeners that are significantly more socially appropriate than baselines. However, many challenges remain, and we release our dataset publicly to spur further progress. See our website for video results, data, and code: https://realtalk.cs.columbia.edu.

Motivation & Objective

• To model the dynamic interplay between verbal and non-verbal communication in dyadic conversations, particularly facial expressions in response to speech.
• To enable control over listener behavior by conditioning on listener-specific factors such as goals, personality, or background.
• To address the lack of high-level semantic reasoning in prior video generation methods that focus only on low-level audio or visual features.
• To create a new, diverse, and unscripted video dataset to support research in multimodal, goal-driven social interaction modeling.
• To demonstrate that LLMs can effectively reason about social context to guide vision-based generation of socially appropriate facial expressions.

Proposed method

• The framework uses a large language model (e.g., GPT-3) to interpret the speaker’s transcript and infer appropriate visual attributes for the listener based on listener-specific conditioning (e.g., relationship goal, personality).
• These inferred visual attributes are then used to retrieve a corresponding facial expression video from a VLM-embedded video retrieval system, such as CLIP.
• The method conditions on both the speaker’s spoken content and the listener’s goal, enabling control over the emotional response without requiring explicit supervision on facial expressions.
• It leverages few-shot prompting in LLMs to generalize across diverse social contexts and infer socially appropriate reactions.
• The system treats listener video generation as a retrieval task, using embeddings from vision-language models to match predicted attributes to real video clips.
• Temporal dynamics are not explicitly modeled, and the method operates on keyframe-level representations, which can lead to timing mismatches in predictions.

Experimental results

Research questions

• RQ1 Can large language models effectively reason about the social context of a conversation to predict appropriate listener facial expressions?
• RQ2 How does conditioning on listener goals or personality improve the social appropriateness of generated facial expressions?
• RQ3 To what extent can a vision-language model retrieve socially appropriate listener videos based on LLM-generated attributes?
• RQ4 How does the proposed method compare to baseline models in generating facial expressions that align with human social expectations?
• RQ5 What are the primary failure modes of the framework, and how do they relate to missing modalities or temporal misalignment?

Key findings

• The proposed method significantly outperforms baseline models, including Learning2Listen, in generating socially appropriate listener facial expressions, as confirmed by human evaluation.
• Human evaluators preferred the system’s predicted 'socially correct' listeners over both the 'socially incorrect' predictions and the Learning2Listen baseline at a statistically significant rate.
• The framework generalizes well to open-world listener goals, such as personality traits and relationship dynamics, correctly inferring appropriate reactions in diverse scenarios.
• Failure cases are primarily due to lack of multimodal speaker context (e.g., facial expressions or prosody), leading to misinterpretation of tone or intent.
• Temporal misalignment is a key limitation, where predicted listener expressions occur before or after the relevant speech event, such as laughing before a punchline.
• Language model hallucinations—especially with metaphors or vague statements—lead to incorrect inferences about speaker intent, resulting in inappropriate listener responses.