[Paper Review] Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation

Navya Martin Kollapally, Christa M. Akers+1 more|arXiv (Cornell University)|Jan 27, 2026

Neurobiology of Language and Bilingualism0 citations

TL;DR

The paper proposes a graph neural network framework that models discourse as a directed multimodal graph of lexical items and gestures to estimate aphasia severity, showing that speech-gesture interactions, not isolated lexical features, encode severity.

ABSTRACT

Aphasia is an acquired language disorder caused by injury to the regions of the brain that are responsible for language. Aphasia may impair the use and comprehension of written and spoken language. The Western Aphasia Battery-Revised (WAB-R) is an assessment tool administered by speech-language pathologists (SLPs) to evaluate the aphasia type and severity. Because the WAB-R measures isolated linguistic skills, there has been growing interest in the assessment of discourse production as a more holistic representation of everyday language abilities. Recent advancements in speech analysis focus on automated estimation of aphasia severity from spontaneous speech, relying mostly in isolated linguistic or acoustical features. In this work, we propose a graph neural network-based framework for estimating aphasia severity. We represented each participant's discourse as a directed multi-modal graph, where nodes represent lexical items and gestures and edges encode word-word, gesture-word, and word-gesture transitions. GraphSAGE is employed to learn participant-level embeddings, thus integrating information from immediate neighbors and overall graph structure. Our results suggest that aphasia severity is not encoded in isolated lexical distribution, but rather emerges from structured interactions between speech and gesture. The proposed architecture offers a reliable automated aphasia assessment, with possible uses in bedside screening and telehealth-based monitoring.

Motivation & Objective

• Motivate automated, holistic aphasia assessment beyond isolated linguistic measures like the WAB-R.
• Investigate whether discourse-level speech-gesture interactions improve aphasia severity estimation.
• Develop a graph-based representation of multimodal discourse and learn participant embeddings.
• Assess the potential for bedside screening and telehealth monitoring using automated methods.

Proposed method

• Represent each participant's discourse as a directed multimodal graph with lexical items and gestures as nodes.
• Encode edges for word-word, gesture-word, and word-gesture transitions to capture transitions and interactions.
• Apply GraphSAGE to learn participant-level embeddings from the graph structure.
• Integrate information from immediate neighbors and overall graph topology to estimate aphasia severity.
• Evaluate the framework as a tool for automated aphasia assessment in clinical and remote settings.

Experimental results

Research questions

• RQ1 Can a directed multimodal graph of speech and gesture better capture aphasia severity than isolated lexical features?
• RQ2 Does GraphSAGE-based embedding of discourse graphs improve severity estimation compared to traditional features?
• RQ3 Are speech-gesture interactions essential for accurate aphasia severity estimation in spontaneous discourse?

Key findings

• Aphasia severity appears to be encoded in structured speech-gesture interactions rather than in isolated lexical distribution.
• The proposed graph-based framework yields a reliable automated aphasia assessment suitable for bedside screening and telehealth monitoring.
• Graph neural network embeddings integrate local neighbor information and global graph structure to estimate severity.