[Paper Review] Hyperparameter Optimization and Boosting for Classifying Facial Expressions: How good can a "Null" Model be?

James Bergstra, David Cox|arXiv (Cornell University)|Jun 14, 2013

Face and Expression Recognition13 references17 citations

TL;DR

This paper presents a hyperparameter optimization and boosting framework using a 'Null model'—a convolutional network with random filters, PCA, pooling, and normalization—achieving 60% accuracy on the Facial Expression Recognition challenge via automated TPE optimization. By ensembling four optimized models, the method reaches 65.5% accuracy, ranking 5th among 56 participants, demonstrating that strong performance can emerge from systematic hyperparameter tuning without novel architectures or backpropagation.

ABSTRACT

One of the goals of the ICML workshop on representation and learning is to establish benchmark scores for a new data set of labeled facial expressions. This paper presents the performance of a "Null" model consisting of convolutions with random weights, PCA, pooling, normalization, and a linear readout. Our approach focused on hyperparameter optimization rather than novel model components. On the Facial Expression Recognition Challenge held by the Kaggle website, our hyperparameter optimization approach achieved a score of 60% accuracy on the test data. This paper also introduces a new ensemble construction variant that combines hyperparameter optimization with the construction of ensembles. This algorithm constructed an ensemble of four models that scored 65.5% accuracy. These scores rank 12th and 5th respectively among the 56 challenge participants. It is worth noting that our approach was developed prior to the release of the data set, and applied without modification; our strong competition performance suggests that the TPE hyperparameter optimization algorithm and domain expertise encoded in our Null model can generalize to new image classification data sets.

Motivation & Objective

• To evaluate the performance of a simple, non-learning-based model (Null model) with random filters and standard preprocessing on a new image classification benchmark.
• To investigate whether automated hyperparameter optimization can outperform expert intuition in model selection for new datasets.
• To develop and apply a novel boosting algorithm (HyperBoost) that combines hyperparameter-optimized base learners into an ensemble for improved generalization.
• To assess whether a model space designed prior to data release can generalize effectively to new image classification tasks.

Proposed method

• A configuration space of 238 conditional hyperparameters was defined for one-, two-, and three-layer convolutional networks with random filters drawn from Gaussian or PCA-based projections.
• The model pipeline included image scaling, affine warping, filterbank normalized cross-correlation, local pooling, di-histogram spatial pooling, and an L2-SVM classifier.
• Hyperopt's Tree-structured Parzen Estimator (TPE) was used to perform Bayesian hyperparameter optimization over the configuration space, selecting 1,000 non-degenerate configurations per round.
• A novel boosting algorithm, HyperBoost, was introduced that treats hyperparameter-optimized models as weak learners, sequentially selecting and combining them to improve test accuracy.
• Ensemble construction used a boosting approach based on hinge loss, iteratively improving test performance while maintaining high training accuracy.
• All experiments were conducted on a single machine with four GPUs, using a slow file system, and configurations were validated for degeneracy before training.

Experimental results

Research questions

• RQ1 Can a 'Null model' with random filters and standard preprocessing achieve competitive performance on a new image classification task without backpropagation or unsupervised pretraining?
• RQ2 To what extent can automated hyperparameter optimization with TPE surpass human intuition in model selection for new datasets?
• RQ3 Can a boosting-based ensemble of hyperparameter-optimized models significantly improve generalization beyond the best individual model?
• RQ4 How well does a model space designed before data release generalize to new image classification benchmarks?
• RQ5 What is the impact of overfitting in high-capacity base learners (e.g., 9,000 features) when combined via boosting?

Key findings

• The Null model with hyperparameter optimization achieved 60% test accuracy on the Facial Expression Recognition challenge, ranking 12th out of 56 participants.
• An ensemble of four models constructed via HyperBoost achieved 65.5% accuracy, ranking 5th overall in the competition.
• The training accuracy of individual models ranged from 85% to 97%, indicating strong overfitting, yet the ensemble maintained high generalization performance.
• The model and training pipeline were designed before the data release and applied without modification, demonstrating strong generalization capability.
• The performance of the ensemble was significantly better than any single model, confirming the effectiveness of HyperBoost in combining optimized base learners.
• The results suggest that hyperparameter optimization and ensemble construction can yield high performance even without advanced deep learning components like backpropagation or RBMs.