[Paper Review] Static Gesture Recognition using Leap Motion

Babak Toghiani-Rizi, Christofer Lind+2 more|arXiv (Cornell University)|May 16, 2017

Hand Gesture Recognition Systems5 references3 citations

TL;DR

This paper presents a static gesture recognition system using the Leap Motion sensor and machine learning to enable an automated bartender interface for ordering drinks and food via hand gestures. Using a k-Nearest Neighbor classifier on normalized 10-dimensional feature vectors derived from fingertip-to-palm distances, the system achieved 95% average accuracy in classifying eight distinct gestures, demonstrating feasibility as a complementary tool to human bartenders despite ethical limitations.

ABSTRACT

In this report, an automated bartender system was developed for making orders in a bar using hand gestures. The gesture recognition of the system was developed using Machine Learning techniques, where the model was trained to classify gestures using collected data. The final model used in the system reached an average accuracy of 95%. The system raised ethical concerns both in terms of user interaction and having such a system in a real world scenario, but it could initially work as a complement to a real bartender.

Motivation & Objective

• To develop a gesture-based ordering system for bars to improve communication in noisy environments.
• To explore the feasibility of using Leap Motion for accurate static hand gesture recognition in a real-world application.
• To evaluate machine learning models for classifying hand gestures with minimal user-specific customization.
• To address practical and ethical challenges in deploying automated gesture-based systems in public service environments.

Proposed method

• Collected 528 gesture samples from ~20 participants, each performing eight distinct gestures three times.
• Extracted 10-dimensional feature vectors using Euclidean distance from each fingertip to its respective palm center.
• Normalized each hand’s 5 finger distances to [0,1] to ensure scale-invariance across hand sizes.
• Reduced feature dimensionality using Principal Component Analysis (PCA) for visualization and analysis.
• Trained and compared three models: k-Nearest Neighbor (k=2), Multilayer Perceptron (MLP), and Multinomial Logistic Regression (MLR) on TensorFlow.
• Implemented real-time feedback using visual and audio cues to enhance user interaction and accessibility.

Experimental results

Research questions

• RQ1 Can Leap Motion accurately capture and classify static hand gestures for a real-time bartender interface?
• RQ2 How does the performance of k-Nearest Neighbor compare to neural network models (MLP and MLR) on a small, real-world gesture dataset?
• RQ3 To what extent does normalization of relative finger distances improve recognition robustness across diverse hand sizes?
• RQ4 What are the key ethical challenges in deploying an automated gesture-based bartender system in public settings?
• RQ5 Can the system function as a viable complement to human bartenders despite limitations in decision-making and inclusivity?

Key findings

• The k-Nearest Neighbor model achieved the highest classification accuracy at 95%, outperforming both the MLP and MLR models on the given dataset.
• Normalization of finger distances relative to each hand’s own range significantly improved model robustness across different hand sizes.
• PCA visualization revealed partial clustering of gesture classes, with Init and Food showing notable overlap, indicating potential for misclassification.
• The system demonstrated strong performance in real-time gesture recognition, supporting practical use in a controlled bar environment.
• Despite high accuracy, the system raises significant ethical concerns, including exclusion of users without ten functional fingers and lack of alcohol sales decision-making capability.
• The open-source implementation is available on GitHub, enabling replication and extension of the work.