[论文解读] Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation

Navya Martin Kollapally, Christa M. Akers+1 more|arXiv (Cornell University)|Jan 27, 2026

Neurobiology of Language and Bilingualism被引用 0

一句话总结

该论文提出一个图神经网络框架，将话语建模为包含词汇项和手势的有向多模态图，以估计失语症严重程度，结果表明语音-手势交互而非孤立的词汇特征才编码了严重程度。

ABSTRACT

Aphasia is an acquired language disorder caused by injury to the regions of the brain that are responsible for language. Aphasia may impair the use and comprehension of written and spoken language. The Western Aphasia Battery-Revised (WAB-R) is an assessment tool administered by speech-language pathologists (SLPs) to evaluate the aphasia type and severity. Because the WAB-R measures isolated linguistic skills, there has been growing interest in the assessment of discourse production as a more holistic representation of everyday language abilities. Recent advancements in speech analysis focus on automated estimation of aphasia severity from spontaneous speech, relying mostly in isolated linguistic or acoustical features. In this work, we propose a graph neural network-based framework for estimating aphasia severity. We represented each participant's discourse as a directed multi-modal graph, where nodes represent lexical items and gestures and edges encode word-word, gesture-word, and word-gesture transitions. GraphSAGE is employed to learn participant-level embeddings, thus integrating information from immediate neighbors and overall graph structure. Our results suggest that aphasia severity is not encoded in isolated lexical distribution, but rather emerges from structured interactions between speech and gesture. The proposed architecture offers a reliable automated aphasia assessment, with possible uses in bedside screening and telehealth-based monitoring.

研究动机与目标

• 推动自动化、整体化的失语评估，超越像 WAB-R 这类孤立语言测量。
• 考察话语层面的语音-手势交互是否提升严重程度估计。
• 建立多模态话语的基于图的表示并学习参与者嵌入。
• 评估在床边筛查和远程健康监测中使用自动化方法的潜力。

提出的方法

• 将每个参与者的话语表示为包含词汇项和手势节点的有向多模态图。
• 对单词-单词、手势-单词、单词-手势转移建立边，以捕捉转移和交互。
• 应用 GraphSAGE 从图结构中学习参与者级嵌入。
• 整合来自直接邻居和整体图拓扑的信息以估计失语症严重程度。
• 将该框架评估为在临床与远程环境中的自动化失语评估工具。

实验结果

研究问题

• RQ1 一个包含语音和手势的有向多模态图是否比孤立的词汇特征更好地捕捉失语症严重程度？
• RQ2 基于 GraphSAGE 的话语图嵌入相比传统特征是否提升严重程度估计？
• RQ3 在自发话语中，语音-手势交互对准确估计失语症严重程度是否至关重要？

主要发现

• 失语症严重程度似乎被结构化的语音-手势交互所编码，而非孤立的词汇分布。
• 所提出的基于图的框架提供了一个可用于床边筛查和远程监测的可靠自动化失语评估工具。
• 图神经网络嵌入将局部邻域信息与全局图结构整合起来以估计严重程度。