[Paper Review] Interpreting social cues to generate credible affective reactions of virtual job interviewers

Hazaël Jones, Nicolas Sabouret+5 more|arXiv (Cornell University)|Feb 20, 2014

Social Robot Interaction and HRI36 references12 citations

TL;DR

This paper presents a real-time pipeline for generating credible affective reactions in a virtual job interviewer by interpreting social cues—such as gestures, gaze, and posture—via a social signal interpretation (SSI) framework. The system computes user communicative performance, updates beliefs about the user’s competence, and drives adaptive emotional responses using a theory-of-mind-inspired model, enhancing realism in virtual job interview training for youth.

ABSTRACT

In this paper we describe a mechanism of generating credible affective reactions in a virtual recruiter during an interaction with a user. This is done using communicative performance computation based on the behaviours of the user as detected by a recognition module. The proposed software pipeline is part of the TARDIS system which aims to aid young job seekers in acquiring job interview related social skills. In this context, our system enables the virtual recruiter to realistically adapt and react to the user in real-time.

Motivation & Objective

• Address the challenge of youth unemployment and social skills deficits among NEETs by providing accessible, private, and repeatable training.
• Develop a virtual recruiter that reacts credibly and dynamically to user behavior during simulated job interviews.
• Enable real-time adaptation of the virtual recruiter’s affective state based on detected nonverbal social cues and performance assessment.
• Improve engagement and learning outcomes in serious games by modeling the recruiter’s beliefs and emotional responses using theory of mind principles.

Proposed method

• Utilizes the SSI (Social Signal Interpretation) framework to detect real-time social cues such as movement energy, gestures, posture, eye gaze, and physiological features.
• Computes a performance index $P_d$ based on the detected cues, categorizing responses as positive (+1), neutral (0), or negative (−1) relative to social expectations.
• Maintains an online user model that updates beliefs $B_{\text{Human}}(\text{topic}_i)$ about the user’s competence using a weighted formula: $B_{\text{Human}}(\text{topic}_i) \leftarrow B_{\text{Human}}(\text{topic}_i) + \alpha \times P_d$.
• Generates desires $D$ based on the agent’s attitude (positive or negative) and the user’s performance, using Algorithm 2 to adjust intentions toward specific topics.
• Applies a goal selection strategy, such as choosing the desire with maximum value, to guide dialogue progression while preserving conversational logic.
• Integrates a theory-of-mind (ToM) approach to simulate the recruiter’s reasoning about the user’s preferences and mental states, enabling more natural and context-sensitive reactions.

Experimental results

Research questions

• RQ1 How can a virtual recruiter generate credible affective reactions in real time during a simulated job interview?
• RQ2 To what extent can social cues—beyond speech—be used to assess a user’s communicative performance in real time?
• RQ3 How can a dynamic user model based on social signal interpretation improve the realism and adaptability of a virtual recruiter?
• RQ4 What role does a theory-of-mind-inspired belief and desire system play in shaping the emotional and strategic behavior of a virtual interviewer?
• RQ5 How can affective responses be systematically computed and updated based on performance indices derived from nonverbal behaviors?

Key findings

• The system successfully computes real-time affective reactions in a virtual recruiter using detected social cues such as gestures, gaze, and posture.
• The performance index $P_d$ enables quantification of user behavior as positive, neutral, or negative, forming the basis for belief and desire updates.
• Belief updates are dynamically computed using a weighted formula with parameter $\alpha$, allowing control over the speed and sensitivity of the recruiter’s perception.
• Desire computation adapts based on the agent’s attitude (positive or negative), ensuring that emotional responses align with social expectations and interaction context.
• The integration of a ToM-inspired model allows the virtual recruiter to reason about the user’s preferences and adjust its behavior accordingly, enhancing interaction credibility.
• The pipeline is designed for extensibility, with plans to incorporate additional cues like expressivity and physiological signals to improve robustness and realism.