[Paper Review] TRB: A Novel Triplet Representation for Understanding 2D Human Body

Haodong Duan, Kwan-Yee Lin+4 more|arXiv (Cornell University)|Oct 25, 2019

Human Pose and Action Recognition43 references4 citations

TL;DR

This paper proposes Triplet Representation for Body (TRB), a novel 2D human body representation combining skeleton keypoints for pose and contour keypoints for shape, enabling richer, more flexible body understanding. The authors introduce TRB-Net, a two-branch network with X-structure, Directional Convolution, and Pairwise Mapping to jointly estimate pose and shape, achieving 13.3% and 15.1% error reduction on skeleton and contour keypoint tasks, respectively, outperforming state-of-the-art methods.

ABSTRACT

Human pose and shape are two important components of 2D human body. However, how to efficiently represent both of them in images is still an open question. In this paper, we propose the Triplet Representation for Body (TRB) -- a compact 2D human body representation, with skeleton keypoints capturing human pose information and contour keypoints containing human shape information. TRB not only preserves the flexibility of skeleton keypoint representation, but also contains rich pose and human shape information. Therefore, it promises broader application areas, such as human shape editing and conditional image generation. We further introduce the challenging problem of TRB estimation, where joint learning of human pose and shape is required. We construct several large-scale TRB estimation datasets, based on popular 2D pose datasets: LSP, MPII, COCO. To effectively solve TRB estimation, we propose a two-branch network (TRB-net) with three novel techniques, namely X-structure (Xs), Directional Convolution (DC) and Pairwise Mapping (PM), to enforce multi-level message passing for joint feature learning. We evaluate our proposed TRB-net and several leading approaches on our proposed TRB datasets, and demonstrate the superiority of our method through extensive evaluations.

Motivation & Objective

• To address the lack of a unified 2D human body representation that captures both pose and shape information.
• To establish a benchmark for TRB estimation, a new task requiring joint learning of pose and shape from 2D images.
• To develop a multi-task deep learning framework that effectively learns skeleton and contour keypoint representations in tandem.
• To demonstrate the utility of TRB in high-level vision tasks such as conditional image generation and human shape editing.

Proposed method

• Propose Triplet Representation for Body (TRB), a compact 2D representation combining skeleton keypoints (for pose) and contour keypoints (for shape).
• Construct large-scale TRB estimation datasets based on LSP, MPII, and COCO by annotating contour keypoints, enabling benchmark evaluation.
• Design TRB-Net, a two-branch network that jointly estimates skeleton and contour keypoints with shared feature learning.
• Introduce X-structure (Xs), a message passing block enabling multi-level feature exchange between skeleton and contour branches.
• Propose Directional Convolution (DC), a task-specific convolutional operator that models inside-out and outside-in spatial relationships between feature maps.
• Implement Pairwise Mapping (PM), a constraint that enforces consistent predictions between skeleton and contour branches to improve generalization.

Experimental results

Research questions

• RQ1 Can a unified 2D human body representation effectively capture both articulated pose and semantic shape information?
• RQ2 How can joint learning of pose and shape improve keypoint estimation accuracy beyond single-task approaches?
• RQ3 To what extent do contour keypoints enhance performance in pose estimation and shape-aware image generation?
• RQ4 Can TRB support practical applications such as conditional image generation and human shape editing?

Key findings

• TRB-Net achieves a 13.3% error reduction on skeleton keypoint estimation and a 15.1% error reduction on contour keypoint estimation compared to state-of-the-art methods.
• On the LSP dataset, TRB-Net achieves 94.5% PCK@0.2, outperforming previous state-of-the-art methods by a significant margin, especially on hard keypoint locations like wrists and ankles.
• On the MPII dataset, TRB-Net achieves 92.2% mean PCKh@0.5, outperforming methods trained only on MPII and remaining competitive even with models using additional data like LIP and LSP.
• With only half the training data from COCO, TRB-Net (hg2-ours_sub) achieves performance comparable to a baseline model trained on the full dataset, demonstrating the efficiency of the TRB representation.
• The TRB-based conditional image generation framework successfully enables pose- and shape-guided image synthesis and explicit human shape editing, outperforming baseline methods in controllability and visual quality.