[Paper Review] A Real-time Hand Gesture Recognition and Human-Computer Interaction System

Pei Xu|arXiv (Cornell University)|Apr 24, 2017

Hand Gesture Recognition SystemsComputer Science22 references62 citations

TL;DR

A real-time gesture-based HCI system using a CNN (modified LeNet-5) on monocular camera input to recognize 16 static gestures with high accuracy, plus Kalman-filtered mouse control and a simple probabilistic response scheme. It also demonstrates ROS-based HRI extension.

ABSTRACT

In this project, we design a real-time human-computer interaction system based on hand gesture. The whole system consists of three components: hand detection, gesture recognition and human-computer interaction (HCI) based on recognition; and realizes the robust control of mouse and keyboard events with a higher accuracy of gesture recognition. Specifically, we use the convolutional neural network (CNN) to recognize gestures and makes it attainable to identify relatively complex gestures using only one cheap monocular camera. We introduce the Kalman filter to estimate the hand position based on which the mouse cursor control is realized in a stable and smooth way. During the HCI stage, we develop a simple strategy to avoid the false recognition caused by noises - mostly transient, false gestures, and thus to improve the reliability of interaction. The developed system is highly extendable and can be used in human-robotic or other human-machine interaction scenarios with more complex command formats rather than just mouse and keyboard events.

Motivation & Objective

• Motivate real-time, low-cost gesture-based HCI using a single monocular camera.
• Develop a CNN-based gesture recognizer that learns features directly from image data.
• Enable stable mouse cursor control via a tracked hand point without extra markers.
• Improve interaction reliability by rejecting transient, false gestures through a simple probabilistic scheme.
• Extend the system to human-robot interaction (HRI) using ROS messages.

Proposed method

• A CNN classifier (modified LeNet-5) processes preprocessed binary hand images to recognize 16 static gestures with high accuracy.
• Hand detection includes background subtraction, color filtering, Gaussian blur, thresholding, morphological operations, contour extraction, and hand region isolation; a distance transform identifies the hand center.
• Polygonal approximation (Ramer-Douglas-Peucker) improves convexity defect detection for fingertip localization.
• The topmost point of palm-based gestures, typically the middle-finger tip, is tracked to drive the mouse cursor; a Kalman filter smooths cursor motion.
• A simple probabilistic discrimination function over a response period prevents the system from reacting to transient/false gestures and stabilizes held actions like drag.
• CNN input sizes of 64x64 yielded best performance at learning rate 0.0001 and momentum 0.9; the model reaches over 99.8% accuracy on the test set.

Experimental results

Research questions

• RQ1 Can a CNN trained on noisy, monocular camera input achieve high-accuracy static gesture recognition with minimal preprocessing?
• RQ2 Can a Kalman-filtered tracking point provide stable, smooth mouse cursor control without markers?
• RQ3 Does a simple probabilistic decision scheme reliably suppress transient false gestures while preserving durative commands?
• RQ4 Can the gesture-based HCI framework be extended to ROS-based human-robot interaction scenarios?

Key findings

• Gesture set comprises 16 static gestures with 19,852 samples collected from five people.
• CNN-based recognition with 64x64 input size achieves over 99.8% accuracy on the test set.
• The Kalman filter improves mouse cursor smoothness during horizontal/vertical motion.
• A simple probabilistic response model effectively rejects transient/false gestures and preserves held actions like drag.
• System demonstrations include keyboard/mouse event triggering and ROS-based control of a simulated robot ( turtle ) via ROS topics.