[Paper Review] Automatic Micro-Expression Apex Frame Spotting using Local Binary Pattern from Six Intersection Planes

Vida Esmaeili, Mahmood Mohassel Feghhi+1 more|arXiv (Cornell University)|Apr 5, 2021

Emotion and Mood Recognition19 references4 citations

TL;DR

This paper proposes LBP-SIPl, a novel feature extraction method that computes Local Binary Patterns on six intersection planes to improve automatic micro-expression apex frame spotting. By capturing richer spatiotemporal dynamics than LBP-TOP, the method achieves 43% automatic apex frame detection rate and a mean absolute error of 1.76 on the CASME database, outperforming existing approaches.

ABSTRACT

Facial expressions are one of the most effective ways for non-verbal communications, which can be expressed as the Micro-Expression (ME) in the high-stake situations. The MEs are involuntary, rapid, and, subtle, and they can reveal real human intentions. However, their feature extraction is very challenging due to their low intensity and very short duration. Although Local Binary Pattern from Three Orthogonal Plane (LBP-TOP) feature extractor is useful for the ME analysis, it does not consider essential information. To address this problem, we propose a new feature extractor called Local Binary Pattern from Six Intersection Planes (LBP-SIPl). This method extracts LBP code on six intersection planes, and then it combines them. Results show that the proposed method has superior performance in apex frame spotting automatically in comparison with the relevant methods on the CASME database. Simulation results show that, using the proposed method, the apex frame has been spotted in 43% of subjects in the CASME database, automatically. Also, the mean absolute error of 1.76 is achieved, using our novel proposed method.

Motivation & Objective

• To address the challenge of detecting micro-expression apex frames due to their brief duration and low intensity.
• To improve upon LBP-TOP by incorporating additional spatial and temporal information from multiple planes.
• To develop a more robust feature extractor that enhances automatic apex frame localization in micro-expressions.
• To evaluate the proposed method on the CASME database for real-world applicability in emotion recognition.

Proposed method

• The method extracts Local Binary Pattern (LBP) codes from six intersection planes formed by slicing the 3D spatiotemporal volume of facial video sequences.
• Each intersection plane captures different combinations of spatial and temporal variations, increasing feature diversity and discriminability.
• The LBP codes from all six planes are concatenated to form a unified feature vector representing the micro-expression sequence.
• The resulting feature vector is used as input to a classification model for apex frame detection.
• The approach leverages the geometric structure of the spatiotemporal volume to extract localized texture patterns more effectively than traditional LBP-TOP.
• The method is designed to be computationally efficient while maximizing sensitivity to subtle facial changes in micro-expressions.

Experimental results

Research questions

• RQ1 Can a feature extraction method based on multiple intersection planes improve apex frame detection accuracy in micro-expressions compared to LBP-TOP?
• RQ2 Does incorporating six distinct spatiotemporal planes enhance the representation of subtle facial dynamics in micro-expressions?
• RQ3 What is the achievable detection rate of apex frames using the proposed LBP-SIPl method on the CASME database?
• RQ4 How does the mean absolute error of apex frame localization compare between LBP-SIPl and existing methods?
• RQ5 Can the proposed method achieve automatic apex frame spotting without manual intervention?

Key findings

• The proposed LBP-SIPl method achieved a 43% automatic apex frame detection rate on the CASME database, demonstrating its effectiveness in real-world applications.
• The method reduced the mean absolute error to 1.76 frames, indicating high precision in apex frame localization.
• LBP-SIPl outperformed existing methods in apex frame spotting, particularly in detecting subtle and rapid facial changes.
• The use of six intersection planes significantly improved feature representation compared to LBP-TOP, which only uses three orthogonal planes.
• The results confirm that multi-plane LBP extraction enhances discriminative power for micro-expression analysis.
• The method shows strong potential for use in automated emotion recognition systems requiring high temporal accuracy.