[Paper Review] Generative Models for Pose Transfer

Patrick Chao, Alexander Li+1 more|arXiv (Cornell University)|Jun 24, 2018

Human Pose and Action Recognition2 references3 citations

TL;DR

This paper proposes a pose transfer pipeline using k-Nearest Neighbors (k-NN) and conditional generative models (pix2pix) to transfer human actions from one person to another in video. The pix2pix model outperforms k-NN in generating sharp, smooth, and artifact-free videos, demonstrating generalization beyond the training pose distribution.

ABSTRACT

We investigate nearest neighbor and generative models for transferring pose between persons. We take in a video of one person performing a sequence of actions and attempt to generate a video of another person performing the same actions. Our generative model (pix2pix) outperforms k-NN at both generating corresponding frames and generalizing outside the demonstrated action set. Our most salient contribution is determining a pipeline (pose detection, face detection, k-NN based pairing) that is effective at perform-ing the desired task. We also detail several iterative improvements and failure modes.

Motivation & Objective

• To develop a robust pipeline for transferring human pose sequences from one individual to another in video form.
• To evaluate the effectiveness of k-NN and conditional GANs (pix2pix) in generating high-quality, temporally consistent video frames.
• To identify and mitigate failure modes such as noisy pose estimation and overfitting to specific body outlines.
• To explore methods for improving generalization, especially to unseen poses not present in training data.
• To reduce computational latency and enable real-time or near real-time pose transfer.

Proposed method

• Used OpenPose to extract 18-joint 2D pose skeletons from input video frames, with median imputation for occluded joints.
• Applied k-NN using L2 distance in joint coordinate space to match poses between source and target individuals.
• Employed a conditional GAN (pix2pix) to directly map pose skeletons to realistic image frames, bypassing intermediate representations.
• Initially included edge detection as a supervisory signal but removed it due to overfitting and reduced generalization.
• Used dlib face detector to supplement pose skeletons with facial contours, improving facial generation quality.
• Explored temporal consistency via frame-by-frame generation with potential for future use of Kalman filters or compressed models.

Experimental results

Research questions

• RQ1 Can k-NN-based pose matching effectively transfer actions between individuals with varying body shapes and clothing?
• RQ2 How does the performance of a conditional GAN (pix2pix) compare to k-NN in generating realistic, sharp, and artifact-free video frames?
• RQ3 To what extent can the generative model generalize to poses not seen during training?
• RQ4 How do noisy pose estimates affect the performance of both k-NN and GAN-based pose transfer?
• RQ5 Can working in joint angle space instead of position space improve k-NN matching robustness across individuals of different heights?

Key findings

• The pix2pix conditional GAN outperformed k-NN in generating sharp, smooth, and realistic video frames with minimal artifacts.
• The pix2pix model successfully generalized to poses not present in the training data, such as raising both arms above the head.
• k-NN performance was severely hampered by noisy pose skeletons, leading to jumpy transitions and poor frame matching.
• Edge detection, while intended to improve supervision, caused overfitting and was ultimately excluded from the final pipeline.
• Using joint angle space instead of joint position space could improve k-NN robustness across individuals with different physical attributes.
• The current pipeline generates frames in ~1 second per frame, far below real-time (60 FPS), indicating a need for optimization.