[论文解读] Human-mediated Large Language Models for Robotic Intervention in Children with Autism Spectrum Disorders

Ruchik Mishra, Karla Conn Welch+1 more|arXiv (Cornell University)|Feb 1, 2024

Autism Spectrum Disorder Research被引用 4

一句话总结

本文提出了一种针对自闭症谱系障碍（ASD）儿童的可扩展机器人干预方法，采用 GPT-2 + BART 流水线，自主生成用于观点采择训练的社会情境、问题及多项选择题选项。NAO 机器人作为刺激者、提示者和强化者，人类评估与 BERTscore 评估均表明，GPT-2 + BART 流水线在生成高质量、临床适宜内容方面优于 GPT-2 + GPT-2。

ABSTRACT

The robotic intervention for individuals with Autism Spectrum Disorder (ASD) has generally used pre-defined scripts to deliver verbal content during one-to-one therapy sessions. This practice restricts the use of robots to limited, pre-mediated instructional curricula. In this paper, we increase robot autonomy in one such robotic intervention for children with ASD by implementing perspective-taking teaching. Our approach uses large language models (LLM) to generate verbal content as texts and then deliver it to the child via robotic speech. In the proposed pipeline, we teach perspective-taking through which our robot takes up three roles: initiator, prompter, and reinforcer. We adopted the GPT-2 + BART pipelines to generate social situations, ask questions (as initiator), and give options (as prompter) when required. The robot encourages the child by giving positive reinforcement for correct answers (as a reinforcer). In addition to our technical contribution, we conducted ten-minute sessions with domain experts simulating an actual perspective teaching session, with the researcher acting as a child participant. These sessions validated our robotic intervention pipeline through surveys, including those from NASA TLX and GodSpeed. We used BERTScore to compare our GPT-2 + BART pipeline with an all GPT-2 and found the performance of the former to be better. Based on the responses by the domain experts, the robot session demonstrated higher performance with no additional increase in mental or physical demand, temporal demand, effort, or frustration compared to a no-robot session. We also concluded that the domain experts perceived the robot as ideally safe, likable, and reliable.

研究动机与目标

• 为解决自闭症谱系障碍儿童机器人干预中的可扩展性缺口，通过大语言模型实现自主内容生成。
• 设计一款社交机器人（NAO），在观点采择治疗会话中充当刺激者、提示者和强化者。
• 通过自动化指标与专家人工评估，评估大语言模型生成的社交情境与问题在质量与临床适宜性方面的表现。
• 比较两种大语言模型流水线——GPT-2 + BART 与 GPT-2 + GPT-2——在文本生成质量与治疗用途中的人类可接受性方面的表现。

提出的方法

• 使用 SOCIALIQA 数据集对 GPT-2 进行微调，实现无监督的社会情境生成。
• 利用第二个大语言模型（BART 或 GPT-2）基于生成的上下文生成问题与多项选择题选项。
• 结合两个模型的损失函数，优化问题与选项的生成，提升 BERTscore 表现。
• 使用 T-SNE 可视化生成的社交情境的语义聚类，以评估其多样性与连贯性。
• 通过四位临床专家进行人工评估，对生成内容的适宜性在 0–10 分制上进行评分，评估年龄适宜性与答案有效性。
• 使用 mnlimicrosoft/deberta-xlarge-mnli 模型计算 BERTscore（F1）以实现对生成文本质量的自动化评估。

Figure 1: NAO robot for teaching perspective-taking to children with ASD. Here ‘Ctx’ refers to the context describing a situation, ‘Q’ is the question based on that context and, A-C are the options consisting of a possible answer.

实验结果

研究问题

• RQ1 大语言模型流水线能否自主生成在社会情境上恰当且具有教育相关性的场景，用于教授自闭症谱系障碍儿童的观点采择能力？
• RQ2 GPT-2 + BART 流水线在生成治疗会话问题与选项方面的表现，相较于 GPT-2 + GPT-2 流水线如何？
• RQ3 专家是否对大语言模型生成的场景与问题给予临床适宜性的评价？
• RQ4 BERTscore 指标与人工评估在生成文本质量上的相关性有多大？

主要发现

• 与 GPT-2 + GPT-2 流水线相比，GPT-2 + BART 流水线在问题与选项生成方面获得了更高的 BERTscore，表明其在语言流畅性与相关性方面表现更优。
• 人工评估显示，具有临床经验的评估者之间具有较强的评分一致性（kappa = 0.68），生成情境的平均适宜性得分为 6.8（满分 10 分）。
• 在选项中至少存在一个有效答案的百分比 agreement 达到 95%，表明答案质量具有高度可靠性。
• BART 基础的流水线在 BERTscore 与人工评估中均优于 GPT-2 基础的流水线，证实其在生成语境准确且教育合理的文本方面具有显著优势。
• T-SNE 可视化结果表明，生成的社交情境形成了连贯且具有语义意义的聚类，支持模型在多样化情境中的一般化能力。

Figure 2: Pipeline of how the robot acts as a stimulator, prompter, and reinforcer. The decisions as whether the response form the child is correct or incorrect is taken by a human evaluator. This pipeline is further described in Section IV .