[Paper Review] Enhanced Touchable Projector-depth System with Deep Hand Pose Estimation

Zhi Chai, Roy Shilkrot|arXiv (Cornell University)|Dec 28, 2018

Tactile and Sensory InteractionsNeuroscience21 references3 citations

TL;DR

This paper proposes a deep learning-enhanced projector-depth system that integrates 3D hand pose estimation with touch detection to improve robustness and functionality in touchable projection interfaces. By training a CNN on top-view hand touch data, the system enables finger-specific touch detection, blob decomposition, and noise-resilient fingertip estimation, significantly enhancing multi-touch interaction and enabling novel applications like a gesture-aware virtual keyboard with finger-based typing assignments.

ABSTRACT

Touchable projection with structured light range cameras is a prolific medium for large interaction surfaces, affording multiple simultaneous users and simple, cheap setup. However robust touch detection in such projector-depth systems is difficult to achieve due to measurement noise. We propose a novel combination of surface touch detection and a deep network for hand pose estimation, which aids in detecting both on- and above-surface hand gestures, disambiguating multiple touch fingers, as well as recovering fingertip positions in face of noisy input. We present the details of our GPU-accelerated system and an evaluation of its performance, as well as applications such as an enhanced virtual keyboard that utilizes the added features.

Motivation & Objective

• To address the challenge of unreliable touch detection in projector-depth systems due to sensor noise and overlapping touch blobs.
• To enable fine-grained, finger-specific touch interaction by integrating deep hand pose estimation into touch detection pipelines.
• To develop a system that supports both on-surface and above-surface gestures with improved robustness to occlusion and measurement noise.
• To enable novel interaction paradigms such as finger-assignment-based virtual keyboards and multi-finger gesture controls.

Proposed method

• A GPU-accelerated processing pipeline combining background modeling, hand segmentation, and deep CNN-based 3D hand pose estimation from top-view depth images.
• A custom-trained deep convolutional neural network (CNN) for 20-joint hand pose estimation, specifically optimized for surface-touch scenarios with high occlusion and noise.
• Pose-aware touch detection that uses estimated hand skeleton to decompose merged touch blobs and assign fingers to specific touch points.
• A large-scale, top-view hand pose dataset collected from real touch interactions, used to train the CNN for improved generalization to noisy depth inputs.
• Integration of pose estimation with blob analysis to disambiguate multiple-finger touches and improve detection accuracy under stereo disparity and sensor noise.
• Application of the system to a novel virtual keyboard where different fingers are assigned to different character sets (lowercase, uppercase, numbers, symbols), enabling gesture-based typing.

Experimental results

Research questions

• RQ1 Can deep hand pose estimation improve touch detection robustness in noisy projector-depth systems?
• RQ2 Can finger-specific touch detection be achieved using pose-aware analysis instead of traditional blob-based methods?
• RQ3 How does integrating 3D hand pose estimation enhance the detection of multi-finger and above-surface gestures?
• RQ4 To what extent does the system improve performance in real-world interaction tasks like virtual keyboard input?
• RQ5 Can the system maintain accuracy under common depth sensor artifacts such as stereo disparity and occlusion?

Key findings

• The system achieved a mean WPM of 12.72 and a CER of 21.63 on a virtual keyboard task with 10 volunteers, demonstrating functional multi-finger typing.
• With auto-completion, the average WPM increased by 57% to 19.98, showing significant performance gains in text entry.
• The system successfully decomposed merged touch blobs into individual finger touches using pose-aware analysis, enabling accurate finger designation.
• Fingertip positions were accurately predicted even under sensor noise and occlusion, improving reliability over traditional connected-component analysis.
• The virtual keyboard prototype enabled novel typing patterns, such as double-letter input via simultaneous multi-finger presses (e.g., 'ff' with index and middle fingers).
• The proposed method demonstrated robustness to depth sensor noise and stereo disparity, outperforming conventional blob-based detection in complex touch scenarios.