[Paper Review] HAKE: Human Activity Knowledge Engine

Yong–Lu Li, Liang Xu+8 more|arXiv (Cornell University)|Apr 13, 2019

Human Pose and Action Recognition24 references39 citations

TL;DR

HAKE builds a large-scale part-state annotated knowledge base linking human body part states to instance activities, enabling a two-stage hierarchical paradigm that improves activity recognition, especially in few-shot settings.

ABSTRACT

Human activity understanding is crucial for building automatic intelligent system. With the help of deep learning, activity understanding has made huge progress recently. But some challenges such as imbalanced data distribution, action ambiguity, complex visual patterns still remain. To address these and promote the activity understanding, we build a large-scale Human Activity Knowledge Engine (HAKE) based on the human body part states. Upon existing activity datasets, we annotate the part states of all the active persons in all images, thus establish the relationship between instance activity and body part states. Furthermore, we propose a HAKE based part state recognition model with a knowledge extractor named Activity2Vec and a corresponding part state based reasoning network. With HAKE, our method can alleviate the learning difficulty brought by the long-tail data distribution, and bring in interpretability. Now our HAKE has more than 7 M+ part state annotations and is still under construction. We first validate our approach on a part of HAKE in this preliminary paper, where we show 7.2 mAP performance improvement on Human-Object Interaction recognition, and 12.38 mAP improvement on the one-shot subsets.

Motivation & Objective

• Bridge instance activities with body part states to improve activity understanding.
• Mitigate long-tail data distribution by decomposing actions into part states.
• Provide an interpretable, linguistically enriched representation via Activity2Vec.
• Demonstrate improvements on HOI benchmarks, especially in few-shot settings.
• Promote broader adoption by inviting community collaboration to enlarge HAKE.

Proposed method

• Annotate part states for 154 instance activities across images using pose-guided part delineation.
• Define ~92 part-state candidates by coordinating human expert judgments via NPMI analysis.
• Develop Part State Classification Network (PSC) to recognize part states with ROI-pooled features and part-level interactiveness priors.
• Create Activity2Vec by fusing visual part-state probabilities with BERT-based linguistic embeddings using a triplet loss to align visual and linguistic spaces.
• Propose a Part States Reasoning Network (PSR) to infer instance activities from part-state embeddings in a hierarchical graph.
• Demonstrate a simplified MNIST-derived analogy to illustrate benefits of part-based decomposition.

Experimental results

Research questions

• RQ1 Can decomposing actions into body part states alleviate the long-tail and data-imbalance issues in large-scale activity recognition?
• RQ2 Does a hierarchical two-stage paradigm (part-state recognition followed by activity reasoning) improve HOI and few-shot recognition performance?
• RQ3 Does combining visual part-state information with linguistic embeddings (Activity2Vec) yield more interpretable and effective activity representations?
• RQ4 How well can part-state based reasoning generalize to complex activities across datasets like HICO/HICO-DET and AVA?
• RQ5 Can HAKE annotations be expanded to support broader tasks such as VQA, action retrieval, and video understanding?

Key findings

• HAKE achieves substantial improvements in HOI recognition benchmarks, including a 7.2 mAP gain over prior methods on HICO under certain settings.
• Using ground-truth part states (HAKE-GT) yields up to 62.5 mAP on HICO, illustrating strong upper-bound potential of the approach.
• On few-shot HOI subsets, HAKE significantly boosts performance, with gains around 11–12 mAP over strong baselines.
• The Part State reasoning and Activity2Vec representations provide improved interpretability and competitive performance, especially when data are scarce.
• HAKE annotates over 7 million body-part state instances across 104k+ images, demonstrating the scale of the resource and its potential for diverse activity understanding tasks.