[Paper Review] HIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices

Jeffrey P. Bigham|arXiv (Cornell University)|Jan 31, 2026

Social Robot Interaction and HRI0 citations

TL;DR

HIDAgent introduces an open-source hardware/software toolkit that lets UI agents observe and control HID-compatible devices by emulating keyboard and mouse, enabling cross-device personal agents separated from the target machines.

ABSTRACT

UI Agents powered by increasingly performant AI promise to eventually use computers the way that people do - by visually interpreting UIs on screen and issuing appropriate actions to control them (e.g., mouse clicks and keyboard entry). While significant progress has been made on interpreting visual UIs computationally, and in sequencing together steps to complete tasks, controlling UIs is still done with system-specific APIs or VNC connections, which limits the platforms and use cases that can be explored. This paper introduces HIDAgent, an open-source hardware/software toolkit enabling UI agents to operate HID-compatible computing systems by emulating the physical keyboard and mouse. HIDAgent is built using three off-the-shelf components costing less than $30 and a Python library supporting flexible integration. We validated the HIDAgent toolkit by building five diverse use case prototypes across mobile and desktop platforms. As a hardware device, HIDAgent supports research into new interaction scenarios where the agents are separated from the devices they control.

Motivation & Objective

• Motivate and define the concept of “Personal Agents” that operate on external hardware separate from control devices.
• Provide an open-source hardware/software toolkit enabling HID-based control of target devices across mobile and desktop platforms.
• Demonstrate the versatility of the approach through diverse prototypes across multiple platforms to explore new interaction scenarios.
• Showcase how hardware separation enables research on trust, cross-device control, and accessibility use cases.

Proposed method

• Describe a low-cost hardware bundle (RP2040, HDMI-to-USB converter, CH340) and a Python library for control.
• Implement HID event emulation on the RP2040 and command handling via UART from a control computer.
• Provide the HIDAgent.py interface with core functions for screenshot capture, mouse/keyboard actions, and helper utilities.
• Offer calibration procedures to map screenshot coordinates to HID coordinates and handle platform-specific settings.
• Expose debugging and logging facilities, including a web-based log viewer and patch_location/gui_diff utilities for scripting.

Figure 1. The HIDAgent toolkit is composed of three off-the-self hardware components ( (i) a RP2040 microcontroller, (ii) an HDMI to USB converter, and (iii) a CH340 USB-to-Serial bridge) and a Python software library that makes it easy to develop programs running on a control computer that observe

Experimental results

Research questions

• RQ1 Can a separate hardware device effectively observe and control HID-compatible targets without installing software on them?
• RQ2 What interaction scenarios (e.g., cross-device control, accessibility tools, data collection) become feasible with HIDAgent?
• RQ3 How well do UI agents perform when operating through an external HID device across mobile and desktop platforms?
• RQ4 What are the practical calibration and platform permission considerations required to deploy HIDAgent across devices?

Key findings

• HIDAgent enables cross-platform control of HID-compatible devices by streaming screen pixels and emitting HID inputs without host software installation.
• Five diverse prototypes demonstrate capabilities like extensible UI agents, universal data collection, screen reader-style accessibility bridging, cross-device interaction, and a helpful observer mode.
• The toolkit supports both on-device processing and remote processing with LLMs, showing viable paths for local and cloud-assisted UI agent workflows.
• Calibration procedures and platform-specific settings are essential for reliable pointer mapping and permissions on iOS, Android, macOS, and Windows.
• The external hardware approach allows researching scenarios where agents are separated from controlled devices, including bring-your-own-agent and secure-control contexts.

Figure 2. HID space to pixel space calibration is done by observing the cursor’s movement in response to HID mouse move events. In this example, the quite faint cursor of the Apple iPhone screen is shown before (i) and after (ii) . The pixel locations that were automatically detected to have changed