[Paper Review] Vision Based Game Development Using Human Computer Interaction

S. Sumathi, S. K. Srivatsa+1 more|arXiv (Cornell University)|Feb 10, 2010

Gaze Tracking and Assistive Technology4 references6 citations

TL;DR

This paper presents a vision-based human-computer interaction system that replaces traditional mouse input with facial gestures, using real-time detection of nose tip movement for mouse cursor control and eye blinks for left/right clicks. Implemented with low-cost USB cameras, the system achieves 30 fps performance and enables natural, hands-free game interaction through facial feature tracking in live video streams.

ABSTRACT

A Human Computer Interface (HCI) System for playing games is designed here for more natural communication with the machines. The system presented here is a vision-based system for detection of long voluntary eye blinks and interpretation of blink patterns for communication between man and machine. This system replaces the mouse with the human face as a new way to interact with the computer. Facial features (nose tip and eyes) are detected and tracked in realtime to use their actions as mouse events. The coordinates and movement of the nose tip in the live video feed are translated to become the coordinates and movement of the mouse pointer on the application. The left or right eye blinks fire left or right mouse click events. The system works with inexpensive USB cameras and runs at a frame rate of 30 frames per second.

Motivation & Objective

• To develop a natural, hands-free human-computer interface for gaming using vision-based interaction.
• To replace traditional mouse input with facial gestures, specifically nose tip movement and eye blinks.
• To enable real-time, low-latency control using inexpensive hardware (USB cameras).
• To support intuitive communication between humans and computers through voluntary eye blink patterns.
• To demonstrate feasibility of vision-based interaction in gaming applications with minimal user training.

Proposed method

• Real-time detection and tracking of facial features—nose tip and eyes—using computer vision techniques in live video streams.
• Translation of nose tip coordinates in the video feed into corresponding mouse pointer movements on the screen.
• Mapping of left or right eye blinks to left or right mouse click events.
• Use of standard USB webcams for input, ensuring low-cost and accessible deployment.
• Implementation of a frame rate of 30 fps to ensure smooth and responsive interaction.
• Application of computer vision and pattern recognition algorithms to interpret facial gestures as input commands.

Experimental results

Research questions

• RQ1 Can facial gestures such as nose movement and eye blinks be reliably detected and translated into mouse control and clicks?
• RQ2 How effective is a low-cost USB camera setup in supporting real-time, interactive gaming via vision-based input?
• RQ3 Can voluntary eye blink patterns be used to encode distinct input commands (e.g., left/right click) with minimal latency?
• RQ4 To what extent does the system enable natural, intuitive human-computer interaction without traditional input devices?
• RQ5 What is the performance ceiling of a vision-based interface in terms of frame rate and responsiveness for gaming applications?

Key findings

• The system successfully maps nose tip movement in real time to mouse pointer movement with a frame rate of 30 fps.
• Left and right eye blinks are accurately detected and mapped to left and right mouse click events, respectively.
• The system operates effectively using only standard, low-cost USB webcams, reducing hardware dependency.
• The interface enables hands-free control of games through natural facial gestures, improving accessibility and user engagement.
• The implementation demonstrates stable and responsive interaction suitable for real-time gaming applications.
• The approach provides a viable alternative to traditional input devices by leveraging facial features as input modalities.