[Paper Review] ROI-Driven Foveated Attention for Unified Egocentric Representations in Vision-Language-Action Systems

Xinhai Sun, Xiang Shi+2 more|arXiv (Cornell University)|Mar 21, 2026

Multimodal Machine Learning Applications0 citations

TL;DR

The paper proposes a deterministic FK-projected ROI workflow using a single external camera to generate hand-centric, egocentric ROIs, enabling cross-robot data reuse and reducing sensor/calibration burden in Vision–Language–Action systems.

ABSTRACT

The development of embodied AI systems is increasingly constrained by the availability and structure of physical interaction data. Despite recent advances in vision-language-action (VLA) models, current pipelines suffer from high data collection cost, limited cross-embodiment alignment, and poor transfer from internet-scale visual data to robot control. We propose a region-of-interest (ROI) driven engineering workflow that introduces an egocentric, geometry-grounded data representation. By projecting end-effector poses via forward kinematics (FK) into a single external camera, we derive movement-aligned hand-centric ROIs without requiring wrist-mounted cameras or multi-view systems. Unlike directly downsampling the full frame, ROI is cropped from the original image before resizing, preserving high local information density for contact-critical regions while retaining global context. We present a reproducible pipeline covering calibration, synchronization, ROI generation, deterministic boundary handling, and metadata governance. The resulting representation is embodiment-aligned and viewpoint-normalized, enabling data reuse across heterogeneous robots. We argue that egocentric ROI serves as a practical data abstraction for scalable collection and cross-embodiment learning, bridging internet-scale perception and robot-specific control.

Motivation & Objective

• Motivate scalable cross-embodiment learning for VLA systems through a reduced, geometry-grounded data representation.
• Introduce a deterministic FK-to-ROI pipeline that generates hand-centric ROIs from a single external camera.
• Provide a governance schema and metadata to ensure reproducibility and cross-robot portability of ROI artifacts.
• Offer an engineering workflow for ROI integration that lowers data-collection and calibration burdens in real deployments.

Proposed method

• Define unified robot base, end-effector, and camera frames with versioned calibration parameters.
• Compute end-effector pose via forward kinematics and project into the external camera using calibrated intrinsics/extrinsics.
• Apply embodiment-aware inward-offset center before cropping to obtain hand-centric ROI patches with zero padding for out-of-frame regions.
• Resize ROIs to a fixed 256x256 resolution and attach ROI confidence metadata.
• Treat ROI as a reproducible derived artifact with a governance schema including versioned metadata for lineage and sharing.
• Describe an ROI-based fusion strategy in VLA architectures by concatenating global and ROI token streams in a ViT framework, thereby biasing attention toward manipulation regions.

Experimental results

Research questions

• RQ1 Can FK-projected ROI from a single external camera provide comparable hand-centric supervision for cross-robot VLA models?
• RQ2 How does a geometry-grounded ROI abstraction impact data collection cost, calibration burden, and reproducibility across heterogeneous robots?
• RQ3 What governance metadata and quality checks are needed to enable reliable cross-embodiment data sharing and regeneration of ROI streams?
• RQ4 How can ROI be integrated with global context and language/proprioception inputs in a unified VLA backbone without architectural changes?
• RQ5 What evaluation protocol can retrofit older datasets to ROI representations and assess transfer robustness across embodiments?

Key findings

• The FK-to-ROI pipeline yields movement-aligned, hand-centric crops with deterministic boundaries and zero padding for out-of-frame regions.
• ROI artifacts are defined with explicit calibration/version metadata to enable reproducible regeneration and governance across sites.
• ROI serves as a foveal supervision channel that preserves local manipulation cues while maintaining global context.
• ROI-based fusion biases attention toward manipulation regions within a multimodal Transformer framework without altering model heads.
• The proposed workflow reduces calibration and sensor burden compared with wrist cameras or multi-view setups, while enabling cross-embodiment transfer of VLA signals.