[Paper Review] Inferring Private Personal Attributes of Virtual Reality Users from Head and Hand Motion Data

Vivek Nair, Christian Räck+9 more|arXiv (Cornell University)|May 30, 2023

Sexuality, Behavior, and Technology9 citations

TL;DR

The paper shows that over 40 private personal attributes can be inferred from VR head/hand motion data alone using Transformer-based models, even in benign gameplay like Beat Saber, highlighting privacy risks in multi-user VR.

ABSTRACT

Motion tracking "telemetry" data lies at the core of nearly all modern virtual reality (VR) and metaverse experiences. While generally presumed innocuous, recent studies have demonstrated that motion data actually has the potential to uniquely identify VR users. In this study, we go a step further, showing that a variety of private user information can be inferred just by analyzing motion data recorded from VR devices. We conducted a large-scale survey of VR users (N=1,006) with dozens of questions ranging from background and demographics to behavioral patterns and health information. We then obtained VR motion samples of each user playing the game "Beat Saber," and attempted to infer their survey responses using just their head and hand motion patterns. Using simple machine learning models, over 40 personal attributes could be accurately and consistently inferred from VR motion data alone. Despite this significant observed leakage, there remains limited awareness of the privacy implications of VR motion data, highlighting the pressing need for privacy-preserving mechanisms in multi-user VR applications.

Motivation & Objective

• Demonstrate that private user attributes can be inferred from VR head and hand motion data in non-adversarial gaming contexts.
• Quantify which attributes are inferable with statistical significance using motion telemetry alone.
• Provide a generalizable machine learning framework for inferring binary attributes from sequential VR motion data.

Proposed method

• Collect motion telemetry (21 features per frame from head and two hands) across 1,006 Beat Saber players who answered a detailed survey.
• Convert 50 selected attributes into binary classification tasks (e.g., age groups, marital status).
• Train transformer-based models on 21×1024 frame sequences to predict binary attributes, using balanced per-class datasets and Monte Carlo cross-validation.
• Evaluate inference success with per-sequence and per-user accuracy and statistical significance via binomial tests (p<0.01, p<0.05).
• Use 100 epochs of training with Adam optimization and BCE loss; report aggregated results across attributes.

Experimental results

Research questions

• RQ1 Can private attributes of VR users be inferred from head and hand motion data alone in non-Adversarial VR games?
• RQ2 Which attributes are inferable with high statistical significance from motion telemetry in Beat Saber?
• RQ3 How effective is a Transformer-based model for this sequential motion data inference task?

Key findings

• Out of 50 attributes, 33 are predicted with high statistical significance (p<0.01) per user, and 8 with moderate significance (p<0.05).
• On a per-sequence basis, 45 of 50 attributes are highly significant (p<0.01) and 1 is moderately significant (p<0.05).
• Macro-significance testing with fictitious inputs yields p<0.0001 for both per-sequence and per-user evaluations, indicating strong overall significance.
• The results hold across a real-world, diverse hardware/software ecosystem, using only head/hand motion data and the weakest adversary model.