[Paper Review] Multi-Granularity Reasoning for Social Relation Recognition from Images

Meng Zhang, Xinchen Liu+4 more|arXiv (Cornell University)|Jan 10, 2019

Human Pose and Action Recognition28 references4 citations

TL;DR

This paper proposes a Multi-Granularity Reasoning (MGR) framework for social relation recognition in images, integrating global scene features, mid-level person-object region representations, and fine-grained pose-guided graphs to bridge the visual-social domain gap. By leveraging pose-aware Person-Object and Person-Pose Graphs with Graph Convolutional Networks, MGR achieves state-of-the-art performance on two public datasets, demonstrating superior accuracy and robustness across diverse relation types.

ABSTRACT

Discovering social relations in images can make machines better interpret the behavior of human beings. However, automatically recognizing social relations in images is a challenging task due to the significant gap between the domains of visual content and social relation. Existing studies separately process various features such as faces expressions, body appearance, and contextual objects, thus they cannot comprehensively capture the multi-granularity semantics, such as scenes, regional cues of persons, and interactions among persons and objects. To bridge the domain gap, we propose a Multi-Granularity Reasoning framework for social relation recognition from images. The global knowledge and mid-level details are learned from the whole scene and the regions of persons and objects, respectively. Most importantly, we explore the fine-granularity pose keypoints of persons to discover the interactions among persons and objects. Specifically, the pose-guided Person-Object Graph and Person-Pose Graph are proposed to model the actions from persons to object and the interactions between paired persons, respectively. Based on the graphs, social relation reasoning is performed by graph convolutional networks. Finally, the global features and reasoned knowledge are integrated as a comprehensive representation for social relation recognition. Extensive experiments on two public datasets show the effectiveness of the proposed framework.

Motivation & Objective

• Address the significant domain gap between visual features and semantic social relations in images.
• Overcome limitations of existing methods that rely on single-granularity representations (e.g., appearance or co-occurrence only).
• Improve recognition accuracy by integrating multi-granularity cues: global scenes, regional person/object features, and fine-grained interactions.
• Model human-object actions and inter-person interactions using pose-guided graph structures for richer relational reasoning.
• Achieve uniform and superior performance across diverse relation types, including ambiguous or low-resource relations.

Proposed method

• Utilize a deep CNN (e.g., ResNet) to extract global scene-level features from the full image.
• Apply an object detector to localize persons and objects, then extract mid-level visual features via CNNs for regional representations.
• Extract human body pose keypoints to guide the construction of two novel graphs: Person-Object Graph (POG) and Person-Pose Graph (PPG).
• Construct the POG by connecting persons to objects based on spatial and pose-aligned interaction cues, modeling actions from persons to objects.
• Build the PPG using keypoint-based connections between paired persons to capture interpersonal interactions.
• Perform relational reasoning via Graph Convolutional Networks (GCNs) on both graphs using message passing to refine representations.

Experimental results

Research questions

• RQ1 Can multi-granularity visual representations—global, regional, and fine-grained—improve social relation recognition beyond single-level features?
• RQ2 How effective is pose-guided graph modeling in capturing meaningful human-object and human-human interactions for relation reasoning?
• RQ3 To what extent do complementary features from global scenes, regional cues, and interaction graphs enhance recognition performance across diverse relation types?
• RQ4 Does integrating multiple granularities lead to more uniform and robust performance, especially on ambiguous or rare relations?
• RQ5 Can graph-based reasoning with GCNs effectively bridge the domain gap between visual input and semantic social relations?

Key findings

• The MGR framework achieves state-of-the-art performance on both the PISA and PIPA datasets, outperforming existing methods in overall accuracy.
• The complete MGR framework with all modules (global, POG, PPG, and late fusion) achieves the highest performance, demonstrating complementary benefits of multi-granularity features.
• The POG with pose guidance significantly outperforms the POG without pose, confirming the importance of action-aware modeling in relation recognition.
• The PPG module improves reasoning for inter-person relations, particularly for intimate relations like friend, family, and couple.
• Global features are more effective for non-intimate relations (e.g., professional, commercial), indicating scene-level cues are more relevant for such categories.
• The method shows uniform performance across all relation types, unlike attention-based models that may be misled by irrelevant contextual objects.