[Paper Review] Egocentric Object Manipulation Graphs

Eadom Dessalene, Michael Maynord+3 more|arXiv (Cornell University)|Jun 5, 2020

Human Pose and Action Recognition34 references17 citations

TL;DR

Ego-OMG proposes a novel graph-based representation for egocentric action anticipation that integrates semantic temporal structure, short-term dynamics, and appearance features via a Graph Convolutional Network (GCN) and CSN features. It achieves state-of-the-art performance, ranking 1st on the unseen test set and 2nd on the seen test set of the EPIC Kitchens Action Anticipation Challenge, with superior long-horizon anticipation due to structured graph modeling of hand-object interactions.

ABSTRACT

We introduce Egocentric Object Manipulation Graphs (Ego-OMG) - a novel representation for activity modeling and anticipation of near future actions integrating three components: 1) semantic temporal structure of activities, 2) short-term dynamics, and 3) representations for appearance. Semantic temporal structure is modeled through a graph, embedded through a Graph Convolutional Network, whose states model characteristics of and relations between hands and objects. These state representations derive from all three levels of abstraction, and span segments delimited by the making and breaking of hand-object contact. Short-term dynamics are modeled in two ways: A) through 3D convolutions, and B) through anticipating the spatiotemporal end points of hand trajectories, where hands come into contact with objects. Appearance is modeled through deep spatiotemporal features produced through existing methods. We note that in Ego-OMG it is simple to swap these appearance features, and thus Ego-OMG is complementary to most existing action anticipation methods. We evaluate Ego-OMG on the EPIC Kitchens Action Anticipation Challenge. The consistency of the egocentric perspective of EPIC Kitchens allows for the utilization of the hand-centric cues upon which Ego-OMG relies. We demonstrate state-of-the-art performance, outranking all other previous published methods by large margins and ranking first on the unseen test set and second on the seen test set of the EPIC Kitchens Action Anticipation Challenge. We attribute the success of Ego-OMG to the modeling of semantic structure captured over long timespans. We evaluate the design choices made through several ablation studies. Code will be released upon acceptance

Motivation & Objective

• To improve action anticipation in egocentric video by modeling semantic temporal structure of human-object interactions.
• To address the limitations of appearance-only or end-to-end video models in long-horizon anticipation.
• To develop a modular framework where appearance features can be easily swapped, enhancing compatibility with existing methods.
• To leverage the consistent egocentric perspective of EPIC Kitchens to model hand-centric cues for improved action prediction.
• To demonstrate that graph-structured modeling of interaction states significantly outperforms sequential or pooling-based aggregation of features.

Proposed method

• Ego-OMG constructs a graph where nodes represent states defined by hand-object contact events, capturing semantic relations between hands and objects over time.
• Graph nodes are embedded into low-dimensional vectors using a Graph Convolutional Network (GCN), enabling structured modeling of interaction dynamics.
• Short-term hand dynamics are modeled via 3D convolutions and trajectory-based contact point anticipation to segment interaction segments.
• Appearance features are extracted using CSN (Convolutional Siamese Networks) with 3D convolutions, capturing spatiotemporal motion and appearance.
• A long short-term memory (LSTM) network processes the sequence of graph-embedded states to model temporal evolution and support long-horizon anticipation.
• The final prediction head classifies future actions based on the LSTM-embedded state history, with ablation studies validating component contributions.

Experimental results

Research questions

• RQ1 How does modeling semantic temporal structure through a hand-object interaction graph improve long-horizon action anticipation compared to end-to-end video models?
• RQ2 What is the contribution of graph embeddings via GCN to action anticipation performance, especially when compared to simple pooling or terminal state classification?
• RQ3 How does the integration of appearance features (CSN) and dynamic hand trajectory modeling enhance anticipation accuracy?
• RQ4 To what extent do pre-trained word embeddings (GloVe) improve the representation of interaction states in the graph?
• RQ5 How does the LSTM-based aggregation of state history compare to alternative aggregation strategies like mean pooling or terminal state classification?

Key findings

• Ego-OMG achieved 1st place on the unseen test set and 2nd on the seen test set of the EPIC Kitchens Action Anticipation Challenge, outperforming all previously published methods.
• The joint model using both CSN and GCN streams outperformed ablations using only one stream, with GCN alone surpassing CSN at a 5-second anticipation horizon.
• LSTM-based aggregation of graph-embedded states achieved 12.81% top-1 accuracy, significantly outperforming mean aggregation (9.74%) and terminal state classification (11.70%).
• Using GCN with GloVe embeddings yielded 12.81% top-1 accuracy, while removing GCN or using identity matrices reduced performance to 6.67% and 3.62%, respectively.
• The ablation study confirmed that graph embeddings and sequential modeling via LSTM are critical for long-horizon anticipation, with performance degrading gracefully as anticipation time increases.
• The modular design allows easy replacement of appearance features, making Ego-OMG complementary to existing action anticipation frameworks.