[Paper Review] GTNet:Guided Transformer Network for Detecting Human-Object Interactions

Adnan Iftekhar, Satish Kumar+3 more|arXiv (Cornell University)|Aug 2, 2021

Multimodal Machine Learning Applications60 references4 citations

TL;DR

GTNet proposes a guided Transformer network that enhances human-object interaction (HOI) detection by fusing relative spatial configurations and object semantics to guide self-attention mechanisms, achieving state-of-the-art performance on V-COCO (58.29 mAP) and HICO-DET datasets through improved contextual feature encoding.

ABSTRACT

The human-object interaction (HOI) detection task refers to localizing humans, localizing objects, and predicting the interactions between each human-object pair. HOI is considered one of the fundamental steps in truly understanding complex visual scenes. For detecting HOI, it is important to utilize relative spatial configurations and object semantics to find salient spatial regions of images that highlight the interactions between human object pairs. This issue is addressed by the novel self-attention based guided transformer network, GTNet. GTNet encodes this spatial contextual information in human and object visual features via self-attention while achieving state of the art results on both the V-COCO and HICO-DET datasets. Code will be made available online.

Motivation & Objective

• To improve human-object interaction detection by effectively leveraging spatial context and object semantics.
• To address the limitation of existing methods in encoding relevant spatial context for HOI prediction.
• To develop a novel guidance mechanism that enhances self-attention in Transformers for visual relationship detection.
• To achieve state-of-the-art performance on standard HOI benchmark datasets, V-COCO and HICO-DET.

Proposed method

• GTNet uses a backbone network to extract visual features from input images and human/object bounding boxes.
• A Guidance Module combines relative spatial configurations and object semantics (via word embeddings) to generate query vectors for attention.
• The TX module applies a Transformer-based self-attention mechanism using guided queries to enrich visual features with spatial context.
• The network employs an early fusion strategy to combine features and predict interactions.
• The guidance mechanism uses either concatenation or element-wise product of spatial and semantic features to refine attention queries.
• A learnable spatial map of size 64×64 is used to encode relative spatial layout, and 1×1 convolutions generate keys and values with 512 channels for optimal performance.

Figure 1: GTNet’s performance with and without its guidance mechanism. Blue indicates GTNet’s predictions with the guidance mechanism; red indicates without. Green bounding boxes indicate the human under consideration; yellow boxes indicate the object. Left column: Simple Scenarios (higher confidenc

Experimental results

Research questions

• RQ1 Can guided self-attention improve HOI detection by encoding spatial and semantic context more effectively?
• RQ2 How does combining object semantics and relative spatial layout affect attention-based feature refinement in HOI tasks?
• RQ3 Does a dedicated guidance mechanism outperform standard attention mechanisms in capturing salient spatial contexts for interactions?
• RQ4 What is the optimal configuration of spatial map size and feature dimensionality for the guidance mechanism?

Key findings

• GTNet achieves a mean average precision (mAP) of 58.29 on the V-COCO test set, setting a new state-of-the-art.
• The model improves by over 6 mAP when the guidance mechanism is included, demonstrating its critical role in performance.
• Semantic guidance alone outperforms spatial guidance, indicating that object category information is more effective than spatial layout alone.
• Using element-wise product for guidance yields better results (58.29 mAP) than concatenation (57.02 mAP), showing improved query refinement.
• Increasing the number of channels for keys and values (up to 512) and spatial map size (up to 64×64) improves performance, with 64×64 and 512 channels selected under GPU memory constraints.
• Qualitative results show that GTNet correctly identifies interactions even in complex scenes, with class activation maps highlighting relevant spatial contexts.

Figure 2: Model Overview. We extract human and object feature vectors from the input feature map, $\mathbf{F}$ via two RRG operations. For clarity we do not explicitly show two separate RRG operations. Human and object features are used to generate a query vector, $\mathbf{f}_{Q}$ . Before feeding $