[Paper Review] Motion-Based Handwriting Recognition

Junshen Kevin Chen, Wanze Xie+1 more|arXiv (Cornell University)|Jan 15, 2021

Hand Gesture Recognition Systems22 references4 citations

TL;DR

This paper proposes motion-based handwriting recognition using inertial sensor data from a smart pen, bypassing the need for visual input or digitizer tablets. By collecting 10,400 sequences of yaw, pitch, roll, and acceleration from 20 users writing lowercase letters, the authors apply data augmentation, autoencoder-based denoising, and train KNN, SVM, CNN, and RNN classifiers—achieving 86.6% accuracy in random split and 53.6% in subject split, with RNN + data augmentation + denoising yielding the best performance.

ABSTRACT

We attempt to overcome the restriction of requiring a writing surface for handwriting recognition. In this study, we design a prototype of a stylus equipped with motion sensor, and utilizes gyroscopic and acceleration sensor reading to perform written letter classification using various deep learning techniques such as CNN and RNNs. We also explore various data augmentation techniques and their effects, reaching up to 86% accuracy.

Motivation & Objective

• To address the challenge of recognizing handwritten characters without visual input or digitizer surfaces, especially in VR/AR or pen-only environments.
• To define and solve a novel problem: classifying handwritten letters using only motion sensor data (yaw, pitch, roll, acceleration) from a smart pen.
• To collect a new, diverse dataset of 10,400 handwritten lowercase letter sequences from 20 users with a custom-built hardware setup.
• To develop a robust pipeline integrating feature interpolation, data augmentation, autoencoder-based denoising, and multiple classifiers for motion-based recognition.
• To evaluate model generalization across users, particularly under subject-split settings where test subjects differ from training subjects.

Proposed method

• Built a custom hardware system using an MPU9250 9-axis IMU, Arduino Uno, and 3D-printed pen mount to capture real-time motion data during writing.
• Collected 20 sequences per lowercase letter from 20 distinct users, resulting in 10,400 total sequences with time-stamped yaw, pitch, roll, and acceleration values.
• Applied feature interpolation to align variable-length sequences, data augmentation to increase training diversity, and a denoising autoencoder to reduce sensor noise.
• Trained four classifiers—KNN, SVM, CNN, and RNN—on the preprocessed motion sequences to predict the written character.
• Evaluated models under two settings: random split (all subjects mixed) and subject split (training and testing on disjoint users), with ablation studies on augmentation and denoising.
• Used confusion matrices and performance tracking to analyze model behavior, particularly overfitting and generalization across writing styles.

Experimental results

Research questions

• RQ1 Can handwriting recognition be effectively performed using only motion sensor data from a pen, without visual input or digitizer surfaces?
• RQ2 How does data augmentation and denoising via autoencoder affect model generalization in motion-based handwriting recognition?
• RQ3 To what extent do different classifiers (KNN, SVM, CNN, RNN) generalize across users with varying handwriting styles?
• RQ4 Does the choice of data split (random vs. subject-based) significantly impact model performance, and why?
• RQ5 What are the primary factors limiting performance, such as sensor data distinctiveness or inter-user writing variability?

Key findings

• The best-performing model, an RNN with data augmentation and denoising autoencoder, achieved 86.6% accuracy in the random split experiment.
• In the subject-split evaluation, the best model reached 53.6% accuracy, indicating significant generalization challenges across different writing habits.
• Data augmentation significantly improved test accuracy across all models, especially in subject-split settings, reducing overfitting.
• Denoising with an autoencoder improved test accuracy by 0.6–1.0 percentage points across models, confirming that sensor noise hinders learning.
• SVM and KNN failed to generalize well in subject-split settings, with KNN achieving only 12.8% accuracy, highlighting limitations of non-temporal models.
• All models exhibited severe overfitting in subject-split experiments, suggesting that writing style variation is a major obstacle to generalization.