[论文解读] Dynamic Multimodal Expression Generation for LLM-Driven Pedagogical Agents: From User Experience Perspective

Ninghao Wan, Jiarun Song+1 more|arXiv (Cornell University)|Mar 10, 2026

Social Robot Interaction and HRI被引用 0

一句话总结

本文提出一种由大模型驱动的方法，为虚拟现实教学代理生成动态多模态表达（语音和手势），显示出改善学习者体验和参与度。

ABSTRACT

In virtual reality (VR) educational scenarios, Pedagogical agents (PAs) enhance immersive learning through realistic appearances and interactive behaviors. However, most existing PAs rely on static speech and simple gestures. This limitation reduces their ability to dynamically adapt to the semantic context of instructional content. As a result, interactions often lack naturalness and effectiveness in the teaching process. To address this challenge, this study proposes a large language model (LLM)-driven multimodal expression generation method that constructs semantically sensitive prompts to generate coordinated speech and gesture instructions, enabling dynamic alignment between instructional semantics and multimodal expressive behaviors. A VR-based PA prototype was developed and evaluated through user experience-oriented subjective experiments. Results indicate that dynamically generated multimodal expressions significantly enhance learners' perceived learning effectiveness, engagement, and intention to use, while effectively alleviating feelings of fatigue and boredom during the learning process. Furthermore, the combined dynamic expression of speech and gestures notably enhances learners' perceptions of human-likeness and social presence. The findings provide new insights and design guidelines for building more immersive and naturally expressive intelligent PAs.

研究动机与目标

• 在VR教学代理中证明自适应且语义对齐的多模态表达的需求。
• 开发一种方法，生成由教学语义引导的协调语音和手势指令。
• 构建一个基于VR的教学代理原型，以评估用户体验。
• 评估动态多模态表达对感知学习效果、参与度、疲劳和社交存在感的影响。

提出的方法

• 提出在语义敏感的提示中协调语音和手势生成。
• 使用LLM驱动的管道将教学语义与多模态表达对齐。
• 实现一个基于VR的教学代理原型。
• 进行以用户体验为导向的主观实验以评估影响。
• 分析对感知学习效果、参与度、疲劳和社交存在感的影响。

实验结果

研究问题

• RQ1 动态生成的多模态表达是否比静态表达提升感知学习效果？
• RQ2 协调的语音和手势表达是否提高学习者的参与度与使用代理的意愿？
• RQ3 动态多模态表达是否缓解VR学习过程中的疲劳和无聊？
• RQ4 综合语音-手势表达如何影响感知的人性化和社交存在感？

主要发现

• 动态生成的多模态表达显著提升学习者的感知学习效果、参与度和使用意愿。
• 动态表达有助于缓解学习过程中的疲劳和无聊。
• 语音与手势的综合动态表达提升对人性化和社交存在感的感知。