[Paper Review] Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos

Shaowei Liu, Subarna Tripathi+2 more|arXiv (Cornell University)|Apr 4, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a novel approach to forecast future hand-object interactions in egocentric videos by jointly predicting hand motion trajectories and interaction hotspots (contact points) on the next active object. Using an Object-Centric Transformer (OCT) with a conditional variational autoencoder head, the model leverages self-attention to reason over hand-object interactions and models uncertainty via probabilistic sampling, achieving state-of-the-art performance on Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+ datasets with fully automated, large-scale data collection.

ABSTRACT

We propose to forecast future hand-object interactions given an egocentric video. Instead of predicting action labels or pixels, we directly predict the hand motion trajectory and the future contact points on the next active object (i.e., interaction hotspots). This relatively low-dimensional representation provides a concrete description of future interactions. To tackle this task, we first provide an automatic way to collect trajectory and hotspots labels on large-scale data. We then use this data to train an Object-Centric Transformer (OCT) model for prediction. Our model performs hand and object interaction reasoning via the self-attention mechanism in Transformers. OCT also provides a probabilistic framework to sample the future trajectory and hotspots to handle uncertainty in prediction. We perform experiments on the Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+ datasets, and show that OCT significantly outperforms state-of-the-art approaches by a large margin. Project page is available at https://stevenlsw.github.io/hoi-forecast .

Motivation & Objective

• To forecast future hand-object interactions in egocentric videos beyond action labels or pixel-level predictions.
• To model uncertainty in future hand motion and object contact points using a probabilistic framework.
• To develop an automatic, large-scale data collection pipeline for hand trajectories and interaction hotspots without human-annotated labels.
• To improve action anticipation performance by leveraging joint trajectory and hotspot prediction as intermediate representations.
• To enable better human-AI collaboration in AR and robotics through accurate, low-dimensional future interaction forecasts.

Proposed method

• An automatic data collection pipeline uses off-the-shelf hand detectors and homography projection to project future hand and contact point locations back to the last observation frame, creating aligned, large-scale training labels.
• The Object-Centric Transformer (OCT) encodes visual features from input frames using a ConvNet, with hand and object features as object-centric tokens and global frame features as context tokens.
• Self-attention in the Transformer encoder enables joint reasoning over hand, object, and environmental context across frames.
• A conditional variational autoencoder (C-VAE) head in the decoder models uncertainty by predicting hand trajectories and interaction hotspots from conditional latent variables derived via cross-attention.
• The model is trained with a reconstruction loss comparing predicted trajectories and hotspots to ground-truth labels from the automatically collected data.
• Fine-tuning on action anticipation tasks uses the OCT encoder with a head MLP, demonstrating transferability of the learned representations.

Experimental results

Research questions

• RQ1 Can joint prediction of hand motion trajectories and interaction hotspots improve future interaction forecasting in egocentric videos compared to action label or pixel prediction?
• RQ2 How can uncertainty in future hand-object interactions be effectively modeled using a probabilistic deep learning framework?
• RQ3 Can large-scale, automatic data collection for trajectories and hotspots replace costly human-annotated datasets without sacrificing performance?
• RQ4 To what extent do trajectory and hotspot predictions benefit downstream tasks like action anticipation?
• RQ5 How does modeling the conditional dependency between hand motion and interaction hotspots enhance prediction accuracy?

Key findings

• The OCT model significantly outperforms state-of-the-art methods on both hand trajectory and interaction hotspot prediction across Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+ datasets.
• The model achieves a 4.3% absolute improvement in verb prediction and a 4.4% improvement in action prediction on EK55, and a 2.9% gain in verb prediction and 2.6% in action prediction on EK100 when fine-tuned for action anticipation.
• The use of probabilistic modeling via C-VAE enables diverse, uncertainty-aware predictions, as shown in qualitative visualizations of multiple plausible futures.
• The automatic data collection pipeline successfully generates large-scale, high-quality labels without human annotation, enabling scalable training.
• Trajectory estimation improves interaction hotspot prediction, and more training data leads to better performance, indicating strong data efficiency.
• Fine-tuning the OCT model pre-trained on trajectory and hotspot prediction yields superior action anticipation performance compared to training from scratch, demonstrating the generalization power of the learned representations.