[Paper Review] Activity Recognition for Autism Diagnosis.

Anish Lakkapragada, Peter Washington+1 more|arXiv (Cornell University)|Aug 18, 2021

Autism Spectrum Disorder Research39 references4 citations

TL;DR

This paper proposes a lightweight LSTM model using temporal hand landmark detection to recognize hand flapping— a key autism-related stimming behavior— from video, achieving 72% validation accuracy and F1 score on the SSBD dataset and generalizing to external videos with under 26,000 parameters, enabling potential deployment in wearable or remote diagnostic systems.

ABSTRACT

A formal autism diagnosis is an inefficient and lengthy process. Families often have to wait years before receiving a diagnosis for their child; some may not receive one at all due to this delay. One approach to this problem is to use digital technologies to detect the presence of behaviors related to autism, which in aggregate may lead to remote and automated diagnostics. One of the strongest indicators of autism is stimming, which is a set of repetitive, self-stimulatory behaviors such as hand flapping, headbanging, and spinning. Using computer vision to detect hand flapping is especially difficult due to the sparsity of public training data in this space and excessive shakiness and motion in such data. Our work demonstrates a novel method that overcomes these issues: we use hand landmark detection over time as a feature representation which is then fed into a Long Short-Term Memory (LSTM) model. We achieve a validation accuracy and F1 Score of about 72% on detecting whether videos from the Self-Stimulatory Behaviour Dataset (SSBD) contain hand flapping or not. Our best model also predicts accurately on external videos we recorded of ourselves outside of the dataset it was trained on. This model uses less than 26,000 parameters, providing promise for fast deployment into ubiquitous and wearable digital settings for a remote autism diagnosis.

Motivation & Objective

• To address the lengthy and inefficient process of formal autism diagnosis, which often results in long delays or missed diagnoses.
• To develop a remote, automated diagnostic tool using digital technologies to detect behavioral markers of autism, particularly stimming behaviors.
• To overcome the challenges of sparse public training data and high video motion variability in detecting hand flapping using computer vision.
• To create a computationally efficient model suitable for deployment in ubiquitous or wearable devices for real-time monitoring.

Proposed method

• The method uses 2D hand landmark detection over time to extract temporal features from video sequences, capturing motion patterns associated with hand flapping.
• These temporal hand landmark sequences are fed into a Long Short-Term Memory (LSTM) recurrent neural network to model sequential behavior patterns.
• The model is trained end-to-end on the Self-Stimulatory Behaviour Dataset (SSBD) to classify videos as containing hand flapping or not.
• Model efficiency is optimized through parameter reduction, resulting in under 26,000 parameters for fast inference on edge devices.
• The approach leverages temporal modeling to handle the sparsity and motion instability common in real-world stimming video data.
• Generalization is evaluated on external videos recorded outside the SSBD, demonstrating robustness beyond the training distribution.

Experimental results

Research questions

• RQ1 Can temporal hand landmark sequences effectively represent hand flapping behavior for classification in low-data regimes?
• RQ2 How well does an LSTM-based model trained on the SSBD generalize to external, unseen video data of hand flapping?
• RQ3 To what extent can a lightweight model with under 26,000 parameters achieve acceptable performance for real-time, wearable autism screening?
• RQ4 Can computer vision techniques overcome challenges of motion noise and data scarcity in detecting autism-related stimming behaviors?

Key findings

• The proposed model achieves a validation accuracy of approximately 72% on the Self-Stimulatory Behaviour Dataset (SSBD) for hand flapping detection.
• The model attains an F1 score of about 72% on the same dataset, indicating balanced precision and recall in binary classification.
• The model generalizes effectively to external videos not included in the training set, demonstrating robustness to distribution shift.
• The final model uses fewer than 26,000 parameters, making it suitable for deployment on resource-constrained wearable or mobile devices.
• Temporal modeling via LSTM effectively captures the dynamic nature of hand flapping despite high motion variability and data sparsity.
• Hand landmark detection over time serves as a reliable and efficient feature representation for identifying autism-related repetitive behaviors.