[Paper Review] Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition

Shizhong Han, Zibo Meng+2 more|arXiv (Cornell University)|Jul 17, 2017

Emotion and Mood Recognition13 references63 citations

TL;DR

The paper introduces Incremental Boosting CNN (IB-CNN) that integrates an incremental boosting layer into CNNs to select discriminative features over time for facial AU recognition, improving generalization with limited data.

ABSTRACT

Recognizing facial action units (AUs) from spontaneous facial expressions is still a challenging problem. Most recently, CNNs have shown promise on facial AU recognition. However, the learned CNNs are often overfitted and do not generalize well to unseen subjects due to limited AU-coded training images. We proposed a novel Incremental Boosting CNN (IB-CNN) to integrate boosting into the CNN via an incremental boosting layer that selects discriminative neurons from the lower layer and is incrementally updated on successive mini-batches. In addition, a novel loss function that accounts for errors from both the incremental boosted classifier and individual weak classifiers was proposed to fine-tune the IB-CNN. Experimental results on four benchmark AU databases have demonstrated that the IB-CNN yields significant improvement over the traditional CNN and the boosting CNN without incremental learning, as well as outperforming the state-of-the-art CNN-based methods in AU recognition. The improvement is more impressive for the AUs that have the lowest frequencies in the databases.

Motivation & Objective

• Address limited AU-coded training data causing CNN overfitting and poor generalization.
• Introduce a boosting mechanism that selects discriminative neurons from the CNN’s lower layer.
• Develop an incremental learning approach to accumulate boosting information across mini-batches.
• Propose a loss function that balances strong- and weak-classifier errors during fine-tuning.
• Demonstrate improved AU recognition on multiple benchmark databases, especially for infrequent AUs.

Proposed method

• Embed an incremental boosting layer in the CNN that selects discriminative FC-layer features (blue nodes) per batch.
• Construct an incremental strong classifier by aggregating current and previous boosted features (blue + red nodes).
• Use AdaBoost-style weighting to update weak learner weights and activate/inactivate neurons (αj and λj).
• Define a composite loss εIB that combines strong-classifier loss and weak-classifier loss to finetune the network.
• Backpropagate the total loss to adjust CNN parameters, enabling information from multiple batches to influence learning.
• Train and evaluate on four AU benchmarks (CK, SEMAINE, BP4D, DISFA) with SGD and momentum.

Experimental results

Research questions

• RQ1 Can incremental boosting within a CNN improve AU recognition when training data are limited?
• RQ2 Does accumulating boosted features across mini-batches reduce overfitting and improve generalization to unseen subjects?
• RQ3 Does the proposed loss that combines strong and weak classifier errors yield better fine-tuning than using strong-classifier loss alone?
• RQ4 Is IB-CNN robust to architectural choices such as the number of FC-layer neurons and learning rate?

Key findings

• IB-CNN outperforms a traditional CNN and a non-incremental boosting CNN (B-CNN) across benchmark AU databases.
• The improvement is especially pronounced for infrequent AUs with limited training samples.
• Using the combined strong+weak loss (εIB) yields better performance than strong-loss-only variants.
• IB-CNN shows robustness to the number of input neurons in the boosting layer and to learning rate choices.