[Paper Review] Action2Vec: A Crossmodal Embedding Approach to Action Learning

Meera Hahn, Andrew Silva+1 more|arXiv (Cornell University)|Jan 2, 2019

Human Pose and Action Recognition38 references42 citations

TL;DR

Action2Vec builds a joint visual-semantic embedding space for actions by combining video-derived spatio-temporal features with verb-based Word2Vec embeddings, enabling zero-shot action recognition and semantic analogy evaluation.

ABSTRACT

We describe a novel cross-modal embedding space for actions, named Action2Vec, which combines linguistic cues from class labels with spatio-temporal features derived from video clips. Our approach uses a hierarchical recurrent network to capture the temporal structure of video features. We train our embedding using a joint loss that combines classification accuracy with similarity to Word2Vec semantics. We evaluate Action2Vec by performing zero shot action recognition and obtain state of the art results on three standard datasets. In addition, we present two novel analogy tests which quantify the extent to which our joint embedding captures distributional semantics. This is the first joint embedding space to combine verbs and action videos, and the first to be thoroughly evaluated with respect to its distributional semantics.

Motivation & Objective

• Motivate a joint embedding space that fuses linguistic cues from action verbs with video-derived spatio-temporal features.
• Develop an end-to-end architecture that maps video clips and associated verbs to a common embedding space.
• Enable zero-shot action recognition and assess semantic structure via analogy tests and distributional semantics.
• Propose evaluation methods that quantify semantic richness of verb embeddings in a multi-modal space.

Proposed method

• Use a Hierarchical Recurrent Neural Network (HRNN) with self-attention to encode video features into a vector.
• Combine a pair of losses: a cosine-based pairwise ranking loss and a cross-entropy classification loss into a dual loss.
• Represent video features via C3D, reduced to 500 dimensions, and process with two-layer HRNN (LSTM 1 and LSTM 2) ending in a 300-dim projection aligned with Word2Vec verb embeddings.
• Incorporate soft attention between feature inputs and LSTMs to capture temporal structure.
• Train using hard negative mining and Adam optimization, with lambda scaling to balance losses.
• Annotate class names to existing Word2Vec verbs and average word embeddings for multi-word class names.

Experimental results

Research questions

• RQ1 Can a joint video-text embedding for actions support accurate zero-shot recognition on standard datasets?
• RQ2 Does Action2Vec preserve semantic similarity between actions that share verbs but differ in nouns?
• RQ3 Do vector arithmetic operations in the Action2Vec space yield coherent novel action descriptors?
• RQ4 How well does the Action2Vec embedding align with distributional semantics from WordNet and Word2Vec for verbs?

Key findings

• Action2Vec achieves state-of-the-art zero-shot action recognition on HMDB51, UCF101, and Kinetics under the reported splits.
• The embedding space semantically clusters visually similar actions, showing locality in the cross-modal space.
• WordNet-based confusion matrices correlate most with Word2Vec, with Action2Vec closely following Word2Vec in semantic structure across datasets.
• Vector arithmetic in Action2Vec demonstrates meaningful analogies, supporting distributional semantic properties for actions.
• Ablation studies show the dual loss and soft attention improve ZSL performance over variants without them.