[Paper Review] JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting

Ron Mokady, Rotem Tzaban+3 more|arXiv (Cornell University)|Jun 17, 2021

Advanced Vision and Imaging43 references4 citations

TL;DR

JOKR introduces a joint keypoint representation for unsupervised cross-domain motion retargeting that disentangles motion from appearance by enforcing domain-invariant keypoint embeddings via a domain confusion loss. The method achieves temporally coherent, visually realistic motion transfer between videos of different shapes and styles—such as animals, flowers, and dancers—without supervision or paired data, using an affine-invariant keypoint representation for robustness to scale and rotation differences.

ABSTRACT

The task of unsupervised motion retargeting in videos has seen substantial advancements through the use of deep neural networks. While early works concentrated on specific object priors such as a human face or body, recent work considered the unsupervised case. When the source and target videos, however, are of different shapes, current methods fail. To alleviate this problem, we introduce JOKR - a JOint Keypoint Representation that captures the motion common to both the source and target videos, without requiring any object prior or data collection. By employing a domain confusion term, we enforce the unsupervised keypoint representations of both videos to be indistinguishable. This encourages disentanglement between the parts of the motion that are common to the two domains, and their distinctive appearance and motion, enabling the generation of videos that capture the motion of the one while depicting the style of the other. To enable cases where the objects are of different proportions or orientations, we apply a learned affine transformation between the JOKRs. This augments the representation to be affine invariant, and in practice broadens the variety of possible retargeting pairs. This geometry-driven representation enables further intuitive control, such as temporal coherence and manual editing. Through comprehensive experimentation, we demonstrate the applicability of our method to different challenging cross-domain video pairs. We evaluate our method both qualitatively and quantitatively, and demonstrate that our method handles various cross-domain scenarios, such as different animals, different flowers, and humans. We also demonstrate superior temporal coherency and visual quality compared to state-of-the-art alternatives, through statistical metrics and a user study. Source code and videos can be found at https://rmokady.github.io/JOKR/ .

Motivation & Objective

• To enable unsupervised motion retargeting between videos of different shapes and styles without requiring object-specific priors or labeled data.
• To disentangle motion content from appearance and shape by learning a shared, domain-invariant keypoint representation across domains.
• To improve temporal coherence and visual quality in cross-domain video generation compared to state-of-the-art methods.
• To enable intuitive control through interpretable, editable keypoint manipulation in generated videos.
• To extend the applicability of motion retargeting to challenging cross-domain pairs, including animals with different proportions and non-human subjects.

Proposed method

• The method employs an encoder-decoder architecture with a shared keypoint head to extract joint keypoint representations (JOKR) from both source and target videos.
• A domain confusion loss is applied to make the keypoint features indistinguishable across domains, enforcing disentanglement of common motion from domain-specific appearance and shape.
• A learned affine transformation is applied to the JOKR to achieve affine invariance, enabling robustness to scale, rotation, and aspect ratio differences between source and target videos.
• Temporal regularization is applied to ensure smooth, coherent motion by constraining keypoint movement between consecutive frames.
• The framework supports intuitive editing by allowing manual manipulation of keypoint positions to generate semantically meaningful motion variations.
• The approach uses a two-stage generation process: first reconstructing the shape from keypoints, then applying the target style to produce the final video frame.

Experimental results

Research questions

• RQ1 Can a joint keypoint representation be learned in an unsupervised manner to enable motion retargeting across videos of different shapes and styles?
• RQ2 How can domain confusion be effectively applied to disentangle common motion from domain-specific appearance and geometry?
• RQ3 To what extent does an affine-invariant keypoint representation improve robustness to scale and rotation differences in cross-domain motion transfer?
• RQ4 Can the learned keypoint representation support intuitive, semantic-level editing of generated video motion?
• RQ5 How does the proposed method compare to state-of-the-art approaches in terms of temporal coherence and visual quality across diverse cross-domain video pairs?

Key findings

• The method achieves superior temporal coherence, with a normalized keypoint displacement of 0.0023 between adjacent frames when using temporal regularization, compared to 0.0055 without it.
• Omitting the domain confusion loss results in keypoint representations that capture shape and texture information, undermining disentanglement and leading to artifacts.
• Removing temporal regularization leads to inconsistent motion and visible artifacts, such as missing body parts, due to erratic keypoint movement.
• The ablation study confirms that the two-stage generation process (shape first, then style) yields higher visual quality than direct keypoint-to-frame generation.
• The learned affine transformation significantly improves training stability and enables successful retargeting of videos with different orientations and scales, such as a horizontal cat and a rotated tiger.
• User studies and quantitative metrics demonstrate that JOKR outperforms state-of-the-art methods in visual quality and temporal consistency across diverse cross-domain pairs, including animals, flowers, and dancers.