[Paper Review] Quaternion Convolutional Neural Networks for Heterogeneous Image Processing

Titouan Parcollet, Mohamed Morchid+1 more|arXiv (Cornell University)|Oct 31, 2018

Neural Networks and ApplicationsComputer Science26 references104 citations

TL;DR

The paper shows that quaternion-valued convolutional encoder-decoders (QCAE) can perfectly learn color dependencies from a single gray-scale image and outperform real-valued CAEs in color reconstruction, with far fewer parameters.

ABSTRACT

Convolutional neural networks (CNN) have recently achieved state-of-the-art results in various applications. In the case of image recognition, an ideal model has to learn independently of the training data, both local dependencies between the three components (R,G,B) of a pixel, and the global relations describing edges or shapes, making it efficient with small or heterogeneous datasets. Quaternion-valued convolutional neural networks (QCNN) solved this problematic by introducing multidimensional algebra to CNN. This paper proposes to explore the fundamental reason of the success of QCNN over CNN, by investigating the impact of the Hamilton product on a color image reconstruction task performed from a gray-scale only training. By learning independently both internal and external relations and with less parameters than real valued convolutional encoder-decoder (CAE), quaternion convolutional encoder-decoders (QCAE) perfectly reconstructed unseen color images while CAE produced worst and gray-scale versions.

Motivation & Objective

• Investigate why QCNNs (via the Hamilton product) improve learning and representation for color images.
• Demonstrate that quaternion-valued networks can learn internal pixel color relations (R,G,B) from grayscale training.
• Compare quaternion-valued encoder-decoder performance to real-valued counterparts on grayscale-to-color reconstruction.
• Quantify gains in reconstruction quality and model efficiency (parameters) under heterogeneous data conditions.

Proposed method

• Use a quaternion-valued convolutional encoder-decoder (QCAE) and a real-valued convolutional encoder-decoder (CAE) with identical topologies.
• Train on a single grayscale image from the Kodak PhotoCD dataset by forming grayscale quaternions for QCAE and grayscale channels for CAE.
• Validate on unseen color images by reconstructing color spaces and comparing to original color images.
• Evaluate reconstructions using PSNR and SSIM metrics, and analyze internal vs external feature learning via the Hamilton product.
• Represent quaternion operations through real-number matrices with split activation functions and a convolution defined by the Hamilton product (Q1 ⊗ Q2).
• Note parameter efficiency: QCAE uses four times more outputs per layer, but overall ~6.4k parameters versus ~25k for CAE.

Experimental results

Research questions

• RQ1 Can quaternion-valued networks learn RGB color dependencies from grayscale training where real-valued networks cannot?
• RQ2 Does the Hamilton product enable better separation of internal (within-pixel) and global (inter-pixel) dependencies in heterogeneous image processing?
• RQ3 Do quaternion encoder-decoders offer superior reconstruction quality with fewer parameters compared to real-valued encoders-decoders?
• RQ4 How do PSNR and SSIM metrics reflect the color-learning capability of QCAE versus CAE on grayscale-to-color reconstruction tasks?

Key findings

• QCAE reconstructs color images almost perfectly from grayscale training, while CAE outputs grayscale (black-and-white) results.
• QCAE achieves PSNR of 31.68 dB and 28.06 dB on two test images, versus CAE PSNR of 29.95 dB and 27.01 dB.
• QCAE SSIM scores are 0.96 and 0.93, while CAE scores are 0.87 and 0.86 for the same tests.
• QCAE uses roughly 6.4k parameters, compared to about 25k for CAE, due to quaternion parameterization and the Hamilton product.
• Results support that quaternion representations separate internal and global dependencies more effectively, enabling robust learning with heterogeneous data.