[Paper Review] All About Knowledge Graphs for Actions

Pallabi Ghosh, Nirat Saini+2 more|arXiv (Cornell University)|Aug 28, 2020

Human Pose and Action Recognition77 references20 citations

TL;DR

This paper proposes a knowledge graph (KG)-based framework for zero-shot and few-shot action recognition using multiple knowledge representations—word embeddings, verb-noun pairs, and visual features—to improve generalization to unseen actions. By constructing hybrid KGs and leveraging graph convolutional networks (GCNs), the method achieves significant performance gains, especially when combining linguistic and visual knowledge, with up to 15.8% improvement in zero-shot accuracy on UCF101 over baseline approaches.

ABSTRACT

Current action recognition systems require large amounts of training data for recognizing an action. Recent works have explored the paradigm of zero-shot and few-shot learning to learn classifiers for unseen categories or categories with few labels. Following similar paradigms in object recognition, these approaches utilize external sources of knowledge (eg. knowledge graphs from language domains). However, unlike objects, it is unclear what is the best knowledge representation for actions. In this paper, we intend to gain a better understanding of knowledge graphs (KGs) that can be utilized for zero-shot and few-shot action recognition. In particular, we study three different construction mechanisms for KGs: action embeddings, action-object embeddings, visual embeddings. We present extensive analysis of the impact of different KGs in different experimental setups. Finally, to enable a systematic study of zero-shot and few-shot approaches, we propose an improved evaluation paradigm based on UCF101, HMDB51, and Charades datasets for knowledge transfer from models trained on Kinetics.

Motivation & Objective

• To investigate the impact of different knowledge graph (KG) construction methods on zero-shot and few-shot action recognition.
• To evaluate the effectiveness of word-based, verb-object-based, and visual feature-based KGs in transferring knowledge from seen to unseen action classes.
• To develop a robust evaluation paradigm that removes common classes between Kinetics (training) and UCF101, HMDB51, and Charades (test) to ensure fair zero-shot and few-shot learning benchmarks.
• To demonstrate that sequence modeling of action phrases (via sentence2vec) improves semantic representation over single-word embeddings for action recognition.

Proposed method

• Construct three distinct knowledge graphs: KG1 using word embeddings of action class names, KG2 using verb-noun pair embeddings from action phrases, and KG3 using visual feature embeddings from few-shot examples.
• Use sentence2vec to encode multi-word action phrases into dense semantic vectors, improving representation over single-word embeddings.
• Build a graph convolutional network (GCN) on each KG to propagate classifier weights from seen to unseen action classes.
• Combine multiple KGs (e.g., KG1 + KG2 + KG3) to create a hybrid knowledge graph that enhances feature transfer in few-shot learning.
• Apply a GCN with 6 layers as the optimal depth for information propagation and smoothing in the graph.
• Design a new evaluation protocol that excludes common classes between Kinetics (training) and UCF101, HMDB51, and Charades (test) to ensure valid zero-shot and few-shot benchmarks.

Experimental results

Research questions

• RQ1 How do different knowledge graph construction methods—word embeddings, verb-noun pairs, and visual features—affect zero-shot and few-shot action recognition performance?
• RQ2 What is the relative contribution of linguistic semantics (via sentence2vec) versus visual features in improving generalization to unseen actions?
• RQ3 Can combining multiple knowledge graph types (e.g., linguistic and visual) lead to better performance than using any single type in few-shot learning?
• RQ4 How does the depth of the graph convolutional network (GCN) influence performance in knowledge transfer for action recognition?
• RQ5 To what extent can GCN-based knowledge transfer outperform simple linear combinations of classifier weights from nearest neighbors?

Key findings

• Using sentence2vec to encode multi-word action phrases significantly improves semantic representation and leads to better zero-shot recognition performance compared to single-word embeddings.
• The combination of KG1 (word-based), KG2 (verb-noun based), and KG3 (visual feature-based) knowledge graphs yields the highest performance, with a 15.8% improvement in zero-shot accuracy on UCF101 over baseline methods.
• KG3 (visual feature-based KG) performs best for actions with distinct visual characteristics, such as "uneven bars" and "shoot gun", where visual cues are highly discriminative.
• The GCN with 6 layers achieves optimal performance, as deeper networks cause over-smoothing and shallower ones limit information propagation.
• The GCN-based approach outperforms simple linear combinations of neighbor classifier weights, demonstrating the value of structured message passing in the graph.
• Visual attention maps (CAM) confirm that the model learns meaningful relationships—e.g., "playing sitar" is activated by "playing guitar" but not by unrelated actions like "biking"—validating the model's generalization capability.