[Paper Review] How Neurotypical and Autistic Children Interact Nonverbally with Anthropomorphic Agents in Open-Ended Tasks

Chuxuan Zhang, Bermet Burkanova+4 more|arXiv (Cornell University)|Mar 8, 2026

Social Robot Interaction and HRI0 citations

TL;DR

The study analyzes spontaneous, open-ended nonverbal interactions of neurotypical and autistic children with six anthropomorphic virtual characters, revealing a rich set of child-specific behaviors and highlighting similarities across groups alongside new behaviors not reported in adult studies.

ABSTRACT

What nonverbal behaviors should a robot respond to? Understanding how children-both neurotypical and autistic-engage with embodied artificial agents is critical for developing inclusive and socially interactive systems. In this paper, we study "open-ended" unconstrained interactions with embodied agents, where little is known about how children behave nonverbally when given few instructions. We conducted a Wizard-of-Oz study in which children were invited to interact nonverbally with 6 different embodied virtual characters displayed on a television screen. We collected 563 (141 unique) nonverbal behaviors produced by children and compare the childre's interaction patterns with those previously reported in an adult study. We also report the presence of repetitive face and hand movements, which should be considered in the development of nonverbally interactive artificial agents.

Motivation & Objective

• Identify similarities and differences in nonverbal interaction patterns between neurotypical and autistic children and adults (for comparison)
• Create a catalog of child-generated nonverbal behaviors relevant to gesture and activity recognition in embodied agents
• Inform inclusive design of socially adaptive embodied systems that can interpret diverse children’s behaviors

Proposed method

• Wizard-of-Oz protocol with six embodied virtual characters (2 humanoids, 2 animals, 2 objects) to enable unconstrained interaction
• Participants: 14 children aged 7–12 (10 neurotypical, 4 autistic); final gesture data from 13 participants after excluding one who drew instead of gesturing
• Data collection via video recordings with two raters and a tie-breaking third rater; additional annotation for 'instructed' vs spontaneous actions
• Software stack included Animaze for character animation, Webcam Motion Capture for upper-body tracking, and Zoom for real-time teleoperation
• Adapted instructions and prompts to be age-appropriate while maintaining consistency with the adult study for comparability
• Analysis mirrored the adult study (zhan2024react) with thematic coding of nonverbal behaviors into physical, emotional, and social categories

Figure 1. Left: Physical setup of the study room. Right: 6 virtual characters used in the study.

Experimental results

Research questions

• RQ1 How do autistic and neurotypical children play or interact with a virtual agent in open-ended tasks?
• RQ2 What nonverbal behaviors do children generate when engaging with embodied agents, and how do these compare to adult data?
• RQ3 Are there child-specific nonverbal behaviors missing from adult datasets that should be considered in HRI/HCI design?
• RQ4 Do developmental or neurodiverse differences emerge in the frequency or type of interactions when engaging with six different agent embodiments?

Key findings

• Children produced 563 interactions in total (537 participant-initiated; 26 agent-initiated) across 89 physical, 18 emotional, and 34 social behaviors
• Compared with adults, children showed a smaller overall variety of nonverbal behaviors but exhibited child-specific actions not found in the adult study (e.g., tongue sticking out, loser sign, imaginative play)
• No significant difference in gesture counts between neurotypical and autistic groups within this small sample
• Significant differences in behavior frequency across characters; social behaviors and emotional behaviors were particularly elevated for the penguin condition after Bonferroni adjustment
• One participant (P7) used drawing and environmental manipulation instead of facial/body gestures, indicating multimodal and context-rich interaction strategies beyond predefined gestures
• Repetitive behaviors were observed in four participants (one neurotypical, three autistic), including motor mannerisms and potential communicative testing, suggesting the need to distinguish communicative vs repetitive actions in agent responses
• Overall, 141 distinct nonverbal behaviors were identified as relevant for designing child-inclusive interactive systems

Figure 2. Interaction behavior counts categorized the type of interaction.