[Paper Review] Project Aria: A New Tool for Egocentric Multi-Modal AI Research

Jakob Julian Engel, Kiran Somasundaram+72 more|arXiv (Cornell University)|Aug 24, 2023

Augmented Reality Applications26 citations

TL;DR

This paper presents Project Aria, a wearable egocentric multi-modal data capture device with software tools and Machine Perception Services to enable research in egocentric perception and personalized AI, along with privacy considerations.

ABSTRACT

Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. These future devices will need to be all-day wearable in a socially acceptable form-factor to support always available, context-aware and personalized AI applications. Our team at Meta Reality Labs Research built the Aria device, an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area. In this paper, we describe the Aria device hardware including its sensor configuration and the corresponding software tools that enable recording and processing of such data.

Motivation & Objective

• Motivate the need for egocentric, multi-modal data to enable context-aware, personalized AI on future AR glasses.
• Introduce the hardware sensor suite, form factor, and recording capabilities of Project Aria.
• Describe the software tools, data formats, and Machine Perception Services that support research using Aria data.
• Outline privacy and responsible innovation principles guiding use of the device and data.
• Demonstrate example research applications enabled by Aria data and services.

Proposed method

• Describe the Project Aria device hardware, sensor configuration, and time-aligned data streams.
• Explain the recording tools, profiles, and VRS data container used for storage and playback.
• Detail Machine Perception Services (MPS) including trajectories, online calibration, semi-dense point clouds, and eye gaze output.
• Present trajectory accuracy (open vs closed loop) and localization robustness under real-world conditions.
• Outline privacy features and responsible innovation principles embedded in hardware and software.
• Show example applications such as lifelong mapping, egocentric scene reconstruction, and activity understanding.

Experimental results

Research questions

• RQ1 What sensor configurations and data-alignment strategies enable robust egocentric perception research?
• RQ2 How can Machine Perception Services derive accurate trajectories, calibrations, and eye gaze from egocentric multi-modal data?
• RQ3 What privacy safeguards are necessary and effective for research with wearable egocentric data?
• RQ4 What research applications become feasible with Project Aria’s multi-modal dataset and tools?

Key findings

• The device provides highly accurate 6-DoF trajectories with open-loop drift below 0.4% of distance traveled and closed-loop RMSE translations typically within 1.5 cm in room-scale scenarios.
• Online calibration accounts for time-varying intrinsics/extrinsics due to temperature and usage, improving geometric accuracy.
• Eye gaze can achieve median ray error around 1.5° after personalized calibration.
• A semi-dense point cloud and trajectories enable intuitive environment understanding from egocentric data.
• Multiple recording profiles balance sensor fidelity with power/bandwidth constraints, facilitating long-term ecological data collection.
• MPS outputs and public datasets accelerate research by providing ready-to-use ego-centric trajectories, calibrations, and gaze data.