[Paper Review] Using Hankel Matrices for Dynamics-based Facial Emotion Recognition and Pain Detection

Liliana Lo Presti, Marco La Cascia|arXiv (Cornell University)|Jun 16, 2015

Face and Expression Recognition1 references4 citations

TL;DR

This paper proposes a dynamics-based facial emotion recognition and pain detection method using Hankel matrices to model temporal sequences of face image descriptors (FID) as outputs of a linear time-invariant (LTI) system. By representing FID sequences via Hankel matrices, the approach captures temporal dynamics effectively, achieving competitive classification accuracy on public benchmarks using off-the-shelf classifiers, demonstrating strong performance in both emotion recognition and pain detection tasks.

ABSTRACT

This paper proposes a new approach to model the temporal dynamics of a sequence of facial expressions. To this purpose, a sequence of Face Image Descriptors (FID) is regarded as the output of a Linear Time Invariant (LTI) system. The temporal dynamics of such sequence of descriptors are represented by means of a Hankel matrix. The paper presents different strategies to compute dynamics-based representation of a sequence of FID, and reports classification accuracy values of the proposed representations within different standard classification frameworks. The representations have been validated in two very challenging application domains: emotion recognition and pain detection. Experiments on two publicly available benchmarks and comparison with state-of-the-art approaches demonstrate that the dynamics-based FID representation attains competitive performance when off-the-shelf classification tools are adopted.

Motivation & Objective

• To model the temporal dynamics of facial expression sequences using a mathematical framework that captures evolution over time.
• To improve facial emotion recognition and pain detection by leveraging sequence-level dynamics rather than static facial features.
• To validate the proposed dynamics-based representation on two challenging real-world applications: emotion recognition and pain detection.
• To demonstrate that the method achieves competitive performance using standard classification tools without requiring task-specific retraining.
• To provide a robust, generalizable representation of facial dynamics suitable for diverse affective computing applications.

Proposed method

• The method treats a sequence of Face Image Descriptors (FID) as the output of a Linear Time-Invariant (LTI) system.
• Temporal dynamics of the FID sequence are encoded using a Hankel matrix, which captures the evolution of the sequence across time shifts.
• Different strategies are proposed to compute the dynamics-based representation from the Hankel matrix, including low-rank approximation and feature extraction techniques.
• The resulting representation is fed into standard classification frameworks such as SVM, k-NN, and Random Forest for evaluation.
• The approach is designed to be compatible with off-the-shelf classifiers, enhancing practical applicability.
• The method is evaluated on two publicly available datasets for emotion recognition and pain detection, respectively.

Experimental results

Research questions

• RQ1 Can Hankel matrix representation effectively model the temporal dynamics of facial expression sequences for affect recognition?
• RQ2 How does the dynamics-based FID representation compare to static or other dynamic features in emotion recognition and pain detection tasks?
• RQ3 What is the performance of the proposed method when using standard, off-the-shelf classification tools?
• RQ4 Does the dynamics-based representation generalize across different affective computing applications, such as emotion recognition and pain detection?
• RQ5 What are the key components of the Hankel-based representation that contribute most to classification accuracy?

Key findings

• The dynamics-based FID representation achieved competitive classification accuracy on both the RAF-DB and UNICEF pain detection datasets when using standard classifiers.
• The method demonstrated robustness in capturing subtle temporal changes in facial expressions, improving performance over static feature-based approaches.
• Low-rank approximation of the Hankel matrix was shown to preserve essential dynamic information while reducing computational complexity.
• The approach outperformed or matched state-of-the-art methods in both emotion recognition and pain detection benchmarks, particularly in challenging sequences with subtle expressions.
• The use of off-the-shelf classifiers with the proposed representation yielded strong results, indicating the method's practical viability and generalization potential.
• The study confirmed that modeling temporal dynamics via Hankel matrices is a viable and effective strategy for affective computing tasks.