[Paper Review] Inferring Dynamic Representations of Facial Actions from a Still Image

Siyang Song, Enrique Sánchez+3 more|arXiv (Cornell University)|Apr 4, 2019

Face recognition and analysis56 references4 citations

TL;DR

This paper proposes a novel image-to-image translation framework that infers dynamic representations (DRs) from still facial images to model short-term temporal dynamics, enabling state-of-the-art performance in Action Unit intensity estimation and dimensional affect estimation without requiring video input. The method uses a self-supervised rank loss to train the network to sort adjacent frames by temporal proximity, capturing bi-directional motion flow around the central image.

ABSTRACT

Facial actions are spatio-temporal signals by nature, and therefore their modeling is crucially dependent on the availability of temporal information. In this paper, we focus on inferring such temporal dynamics of facial actions when no explicit temporal information is available, i.e. from still images. We present a novel approach to capture multiple scales of such temporal dynamics, with an application to facial Action Unit (AU) intensity estimation and dimensional affect estimation. In particular, 1) we propose a framework that infers a dynamic representation (DR) from a still image, which captures the bi-directional flow of time within a short time-window centered at the input image; 2) we show that we can train our method without the need of explicitly generating target representations, allowing the network to represent dynamics more broadly; and 3) we propose to apply a multiple temporal scale approach that infers DRs for different window lengths (MDR) from a still image. We empirically validate the value of our approach on the task of frame ranking, and show how our proposed MDR attains state of the art results on BP4D for AU intensity estimation and on SEMAINE for dimensional affect estimation, using only still images at test time.

Motivation & Objective

• To model temporal dynamics of facial actions from still images, where no explicit video or temporal data is available.
• To develop a self-supervised training approach that does not require predefined target representations for dynamic representation learning.
• To enhance facial action unit (AU) intensity estimation and dimensional affect estimation using inferred dynamic representations from single images.
• To validate the effectiveness of multi-scale dynamic representations (MDR) in capturing short-term motion patterns for improved performance.

Proposed method

• Proposes an image-to-image translation network that generates a dynamic representation (DR) from a single still image, designed as a 3-channel spatial kernel summarizing motion around the central frame.
• Employs a rank loss function to train the network to sort neighboring frames in temporal order based on their distance to the central image, enabling bi-directional temporal flow modeling.
• Introduces a multi-scale dynamic representation (MDR) approach that infers DRs for multiple temporal window lengths, capturing dynamics at different time scales.
• Uses a self-supervised training paradigm where the network learns to rank frames without explicit target representations, improving generalization to unseen images.
• Combines the original image with the inferred DRs to enhance feature learning, demonstrating that appearance and dynamic cues are complementary.
• Applies the DRs as input to CNN-based networks for AU intensity and affect estimation, showing improved performance over single-image baselines.

Experimental results

Research questions

• RQ1 Can dynamic representations be effectively inferred from a single still image to model facial action dynamics?
• RQ2 Does self-supervised training with a rank loss function enable better generalization than methods relying on predefined target representations?
• RQ3 Can multi-scale dynamic representations (MDR) improve performance in AU intensity and dimensional affect estimation tasks using only still images?
• RQ4 How effective is the inferred DR in sorting unseen adjacent frames in temporal order?
• RQ5 To what extent do dynamic representations enhance still-image-based facial behavior analysis compared to appearance-only features?

Key findings

• The proposed MDR approach achieves state-of-the-art performance on the BP4D dataset for Action Unit intensity estimation, with a mean ICC of 0.72, surpassing prior methods that use video input.
• On the SEMAINE dataset, the MDR-based approach achieves a Pearson correlation coefficient (PCC) of 0.335 for arousal and 0.316 for valence, outperforming all compared methods using only still images.
• The self-supervised rank loss training scheme generalizes better than models trained with predefined target representations, as shown by improved performance on unseen images.
• Combining the original image with the MDR representation yields better results than using either the image or DR alone, confirming the complementary nature of appearance and dynamic cues.
• The inferred DRs effectively sort adjacent frames in temporal order, validating their capacity to capture short-term motion dynamics.
• The SDR (single dynamic representation) approach achieves performance comparable to using a single image, but MDR significantly improves performance, demonstrating the value of multi-scale temporal modeling.