[Paper Review] NFS: A Hand Gesture Recognition Based Game Using MediaPipe and PyGame

Md Rafid Islam, Ratun Rahman+2 more|arXiv (Cornell University)|Apr 23, 2022

Hand Gesture Recognition Systems4 citations

TL;DR

This paper presents NFS, a hand gesture recognition game using MediaPipe for real-time hand landmark detection and PyGame for game rendering. By replacing traditional controllers with gesture-based interaction, the system achieves higher fun and replayability despite increased difficulty, demonstrating the viability of gesture-based gaming in human-computer interaction.

ABSTRACT

This paper represents a game which interacts with humans via hand gesture movement. Nowadays, apps like this seem rare, and there seems to be a window opening for this kind of application to be more prevalent and useful in the near future. This application is based on hand gesture movement instead of being dependent on a keyboard and mouse. The main issue was to figure out how to utilize machine learning to make this application work as it should be. First, two games were selected one with a traditional controller and another with hand gesture method. Then these two games based on the difficulty to use, fun elements, gameplay, and replayability were compared. Though the difficulty increases but the other three aspects improve significantly. After going through all of that a conclusion can be drawn that people are more likely to play a simple hand gesture-based game.

Motivation & Objective

• To develop a gesture-based gaming application that replaces traditional input methods like keyboard and mouse.
• To evaluate the usability and user experience of a hand gesture-controlled game compared to a traditional controller-based version.
• To investigate whether gesture-based interaction can improve fun and replayability in gaming despite higher difficulty.
• To demonstrate the practical integration of MediaPipe's hand tracking with PyGame for real-time interactive applications.

Proposed method

• MediaPipe Hands is used to detect 21 keypoint landmarks on the human hand in real time from a video stream.
• The detected hand landmarks are processed to recognize specific gestures, such as finger counting or hand shapes, used as game controls.
• A custom game environment is built using PyGame, which renders the game and interprets gesture inputs for in-game actions.
• The system compares two game versions: one using traditional input and another using hand gestures, with both evaluated on gameplay, difficulty, fun, and replayability.
• The game logic is implemented to respond to gesture states, enabling actions like movement or selection based on hand pose.
• The entire pipeline runs in real time on a standard CPU, ensuring low-latency interaction for responsive gameplay.

Experimental results

Research questions

• RQ1 How does gesture-based interaction compare to traditional controller input in terms of gameplay enjoyment and user engagement?
• RQ2 Can a hand gesture recognition system achieve sufficient accuracy and responsiveness for real-time gaming applications?
• RQ3 What is the trade-off between increased difficulty and improved fun and replayability in gesture-controlled games?
• RQ4 To what extent can MediaPipe's hand tracking be effectively integrated with PyGame for interactive game development?

Key findings

• The gesture-based game version showed significantly higher levels of fun and replayability compared to the traditional controller version.
• Although the gesture-based game had higher difficulty, users reported greater engagement and enjoyment due to intuitive physical interaction.
• The integration of MediaPipe and PyGame enabled real-time, low-latency gesture recognition suitable for interactive gaming.
• The system demonstrated that hand gesture recognition can be a viable alternative to traditional input methods in gaming applications.