[Paper Review] Learning Pain from Action Unit Combinations: A Weakly Supervised Approach via Multiple Instance Learning

Zhanli Chen, Rashid Ansari+1 more|arXiv (Cornell University)|Dec 5, 2017

Emotion and Mood Recognition17 references3 citations

TL;DR

This paper proposes a weakly supervised, two-stage approach for automated pain detection using facial action unit (AU) combinations via multiple instance learning (MIL) and multiple clustered instance learning (MCIL). By leveraging frame-level AU detection and modeling pain as a sequence-level label, the method achieves 87% accuracy and 0.94 AUC on the UNBC-McMaster dataset, demonstrating strong performance and generalization to long clinical videos.

ABSTRACT

Patient pain can be detected highly reliably from facial expressions using a set of facial muscle-based action units (AUs) defined by the Facial Action Coding System (FACS). A key characteristic of facial expression of pain is the simultaneous occurrence of pain-related AU combinations, whose automated detection would be highly beneficial for efficient and practical pain monitoring. Existing general Automated Facial Expression Recognition (AFER) systems prove inadequate when applied specifically for detecting pain as they either focus on detecting individual pain-related AUs but not on combinations or they seek to bypass AU detection by training a binary pain classifier directly on pain intensity data but are limited by lack of enough labeled data for satisfactory training. In this paper, we propose a new approach that mimics the strategy of human coders of decoupling pain detection into two consecutive tasks: one performed at the individual video-frame level and the other at video-sequence level. Using state-of-the-art AFER tools to detect single AUs at the frame level, we propose two novel data structures to encode AU combinations from single AU scores. Two weakly supervised learning frameworks namely multiple instance learning (MIL) and multiple clustered instance learning (MCIL) are employed corresponding to each data structure to learn pain from video sequences. Experimental results show an 87% pain recognition accuracy with 0.94 AUC (Area Under Curve) on the UNBC-McMaster Shoulder Pain Expression dataset. Tests on long videos in a lung cancer patient video dataset demonstrates the potential value of the proposed system for pain monitoring in clinical settings.

Motivation & Objective

• Address the challenge of limited labeled pain data in clinical facial expression analysis by leveraging weakly supervised learning.
• Improve pain detection accuracy by modeling pain as a combination of action units (AUs) rather than relying on individual AU detection alone.
• Develop a decoupled framework that mimics human FACS coding: first detect AUs at the frame level, then infer pain at the sequence level.
• Enable effective pain monitoring in long clinical videos by adapting the model to real-world patient data beyond standard datasets.
• Demonstrate generalization and robustness of the system on trans-dataset evaluation using lung cancer patient videos

Proposed method

• Use state-of-the-art automated facial expression recognition (AFER) tools to detect individual AUs at the frame level from video sequences.
• Construct two novel data structures—Compact-MIL and Clustered-MCIL—to encode AU combinations from frame-level AU scores.
• Apply multiple instance learning (MIL) and multiple clustered instance learning (MCIL) to model pain detection as a weakly supervised problem, where labels are assigned at the sequence level.
• Employ a Gaussian Mixture (GM) approximation in the MCIL framework to enhance feature sparsity and improve margin separation.
• Train the system on the UNBC-McMaster Shoulder Pain Expression dataset with frame-level AU annotations and sequence-level pain labels.
• Test the system on long video subsequences from a lung cancer patient dataset, comparing predictions with human expert consensus on AU combinations

Experimental results

Research questions

• RQ1 Can modeling pain as combinations of action units (AUs) improve detection accuracy compared to individual AU-based approaches?
• RQ2 How effective is weakly supervised learning via MIL and MCIL in detecting pain when only sequence-level labels are available?
• RQ3 Does the proposed decoupled framework—AU detection followed by AU combination modeling—achieve better performance than end-to-end binary pain classifiers?
• RQ4 Can the system generalize to long clinical videos and maintain consistency with human expert annotations across diverse patient populations?
• RQ5 To what extent does feature sparsity from clustered representations improve model performance and alignment with human coding decisions?

Key findings

• The proposed system achieved 87% pain recognition accuracy and 0.94 AUC on the UNBC-McMaster Shoulder Pain Expression dataset using the Compact-MIL framework.
• The Clustered-MCIL variant outperformed Compact-MIL in AUC (0.94 vs. 0.92), attributed to improved feature sparsity and margin separation.
• On long video subsequences from a lung cancer patient dataset, the system showed 82.9% consistency with human coders who identified pain in at least two out of three experts.
• The system achieved 68.6% consistency with the single coder on less credible cases, indicating robustness despite annotation ambiguity.
• The system demonstrated strong generalization to unseen clinical data, with 88.9% consistency on negative (no pain) cases across all three coders.
• The decoupled structure—separating AU detection from pain inference—improved performance by enabling better handling of uncertainty and latent pain representation