[Paper Review] Classification of fused face images using multilayer perceptron neural network

Debotosh Bhattacharjee, Mrinal Kanti Bhowmik+3 more|arXiv (Cornell University)|Jul 5, 2010

Face and Expression Recognition17 references7 citations

TL;DR

This paper proposes a fusion-based face recognition method that combines visual and thermal face images using pixel-level fusion to enhance robustness against illumination, pose, and expression variations. The fused images are projected into eigenspace and classified via a multilayer perceptron, achieving a 95.07% recognition success rate on the OTCBVS database, demonstrating significant performance improvement over individual modalities.

ABSTRACT

This paper presents a concept of image pixel fusion of visual and thermal faces, which can significantly improve the overall performance of a face recognition system. Several factors affect face recognition performance including pose variations, facial expression changes, occlusions, and most importantly illumination changes. So, image pixel fusion of thermal and visual images is a solution to overcome the drawbacks present in the individual thermal and visual face images. Fused images are projected into eigenspace and finally classified using a multi-layer perceptron. In the experiments we have used Object Tracking and Classification Beyond Visible Spectrum (OTCBVS) database benchmark thermal and visual face images. Experimental results show that the proposed approach significantly improves the verification and identification performance and the success rate is 95.07%. The main objective of employing fusion is to produce a fused image that provides the most detailed and reliable information. Fusion of multiple images together produces a more efficient representation of the image.

Motivation & Objective

• To improve face recognition performance under challenging conditions such as illumination changes, pose variations, and facial expression changes.
• To address limitations in standalone visual or thermal face recognition by fusing complementary information from both modalities.
• To develop a robust image fusion technique that produces a single, detailed, and reliable representation for improved classification.
• To evaluate the effectiveness of multilayer perceptron (MLP) neural networks in classifying fused face images.
• To demonstrate that pixel-level fusion of visual and thermal images enhances verification and identification accuracy.

Proposed method

• Fusing visual and thermal face images at the pixel level to create a composite image with enhanced structural and thermal information.
• Projecting the fused images into an eigenspace using principal component analysis (PCA) for dimensionality reduction and feature extraction.
• Training a multilayer perceptron (MLP) neural network on the reduced eigenspace features for classification.
• Using the OTCBVS database as the benchmark dataset containing synchronized visual and thermal face images.
• Applying standard preprocessing steps including normalization and alignment prior to fusion and feature extraction.
• Optimizing the MLP architecture for binary and multi-class face recognition tasks using backpropagation.

Experimental results

Research questions

• RQ1 Can pixel-level fusion of visual and thermal face images improve recognition performance under adverse conditions?
• RQ2 How does the combination of visual and thermal data affect the discriminative power of face features?
• RQ3 To what extent does the use of an MLP classifier enhance classification accuracy on fused face images?
• RQ4 What is the impact of eigenspace projection on the generalization and efficiency of the recognition system?
• RQ5 How does the proposed fusion approach compare to individual visual or thermal face recognition systems?

Key findings

• The proposed fusion method achieved a recognition success rate of 95.07% on the OTCBVS benchmark database.
• Fused images provided more detailed and reliable information than either visual or thermal images alone.
• The combination of pixel-level fusion and MLP classification significantly outperformed individual modality recognition systems.
• The eigenspace projection effectively reduced dimensionality while preserving discriminative features for classification.
• The method demonstrated robustness to illumination variations, pose changes, and facial expression differences.
• The results confirm that multimodal fusion enhances the overall performance of face recognition systems.