[Paper Review] Virtual Mouse And Assistant: A Technological Revolution Of Artificial Intelligence

Jagbeer Singh, Yash Goel+2 more|arXiv (Cornell University)|Mar 11, 2023

AI in Service Interactions4 citations

TL;DR

This paper presents a novel human-computer interaction system integrating a virtual mouse and AI-powered assistant that leverages computer vision to enable gesture-based cursor control and task automation. Using a single camera, the system maps index and middle finger gestures to mouse movements and clicks, while extending AI assistant capabilities beyond basic searches to include full media control and automated web interactions on platforms like Google and YouTube.

ABSTRACT

The purpose of this paper is to enhance the performance of the virtual assistant. So, what exactly is a virtual assistant. Application software, often called virtual assistants, also known as AI assistants or digital assistants, is software that understands natural language voice commands and can perform tasks on your behalf. What does a virtual assistant do. Virtual assistants can complete practically any specific smartphone or PC activity that you can complete on your own, and the list is continually expanding. Virtual assistants typically do an impressive variety of tasks, including scheduling meetings, delivering messages, and monitoring the weather. Previous virtual assistants, like Google Assistant and Cortana, had limits in that they could only perform searches and were not entirely automated. For instance, these engines do not have the ability to forward and rewind the song in order to maintain the control function of the song; they can only have the module to search for songs and play them. Currently, we are working on a project where we are automating Google, YouTube, and many other new things to improve the functionality of this project. Now, in order to simplify the process, we've added a virtual mouse that can only be used for cursor control and clicking. It receives input from the camera, and our index finger acts as the mouse tip, our middle finger as the right click, and so forth.

Motivation & Objective

• To overcome limitations of existing virtual assistants like Google Assistant and Cortana, which lack full automation and media control capabilities.
• To enhance human-computer interaction by enabling intuitive, gesture-based control using hand gestures as input.
• To develop a system that supports automated interaction with web applications such as YouTube and Google, including media playback control.
• To integrate a virtual mouse function that maps finger gestures to cursor movement and clicks using real-time computer vision.
• To create a unified interface combining AI assistant functionality with direct hardware-like control for improved user efficiency.

Proposed method

• Utilizes a single camera to capture real-time video input for hand gesture detection.
• Employs computer vision algorithms to identify and track the index finger as the virtual mouse pointer.
• Maps the middle finger to simulate right-click functionality, enabling context menu access.
• Integrates AI assistant modules to interpret natural language voice commands and perform automated tasks.
• Extends assistant functionality beyond search to include media control (play, pause, forward, rewind) on YouTube and Google.
• Combines gesture recognition with NLP-driven task automation to enable end-to-end control of web applications.

Experimental results

Research questions

• RQ1 How can hand gesture recognition be effectively used to replace traditional mouse input in a virtual environment?
• RQ2 To what extent can a virtual assistant be enhanced to support full media control beyond basic search and playback?
• RQ3 Can a unified system combining gesture-based navigation and AI-driven task automation improve user interaction efficiency?
• RQ4 What are the technical challenges in enabling real-time, accurate mapping of finger gestures to mouse actions using a single camera?
• RQ5 How does the integration of AI assistant functions with virtual mouse control affect overall system usability and responsiveness?

Key findings

• The system successfully maps index finger movement to virtual mouse cursor control with real-time responsiveness using computer vision.
• Right-click functionality is accurately simulated using middle finger gestures, enabling context menu access without physical input devices.
• The AI assistant component extends beyond search to support media control, including play, pause, forward, and rewind on YouTube and Google.
• The integration of gesture-based navigation with AI task automation enables end-to-end control of web applications without manual keyboard or mouse input.
• The system demonstrates improved usability for hands-free interaction, particularly in scenarios requiring continuous media or application control.
• The approach enables a seamless, intuitive interface that reduces reliance on traditional input devices while maintaining high task automation fidelity.