[Paper Review] Deep Complex Networks

Chiheb Trabelsi, Olexa Bilaniuk+8 more|PolyPublie (École Polytechnique de Montréal)|May 27, 2017

Music and Audio Processing29 references166 citations

TL;DR

The paper develops a complete set of building blocks for complex-valued deep neural networks, including complex convolution, complex batch normalization, and complex activations, and demonstrates competitive performance on vision and audio tasks such as CIFAR, MusicNet, and TIMIT.

ABSTRACT

At present, the vast majority of building blocks, techniques, and architectures for deep learning are based on real-valued operations and representations. However, recent work on recurrent neural networks and older fundamental theoretical analysis suggests that complex numbers could have a richer representational capacity and could also facilitate noise-robust memory retrieval mechanisms. Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models. In this work, we provide the key atomic components for complex-valued deep neural networks and apply them to convolutional feed-forward networks and convolutional LSTMs. More precisely, we rely on complex convolutions and present algorithms for complex batch-normalization, complex weight initialization strategies for complex-valued neural nets and we use them in experiments with end-to-end training schemes. We demonstrate that such complex-valued models are competitive with their real-valued counterparts. We test deep complex models on several computer vision tasks, on music transcription using the MusicNet dataset and on Speech Spectrum Prediction using the TIMIT dataset. We achieve state-of-the-art performance on these audio-related tasks.

Motivation & Objective

• Provide a general formulation for complex-valued deep neural networks and their building blocks.
• Apply complex-valued operations to convolutional nets and LSTMs.
• Demonstrate competitive performance on real-world tasks across vision and audio datasets.

Proposed method

• Represent complex numbers via paired real/imaginary feature maps.
• Derive complex convolution as a real-valued operation on split real/imag components.
• Introduce complex batch normalization via whitening of 2D real-imaginary vectors.
• Propose complex weight initialization using magnitude distribution (Rayleigh) and phase randomization.
• Evaluate with activation functions including C-ReLU, modReLU, and z-ReLU across tasks.
• Compare complex networks against real-valued counterparts on CIFAR-10/100, SVHN*, MusicNet, and TIMIT.

Experimental results

Research questions

• RQ1 Can complex-valued networks match or exceed real-valued architectures on standard vision benchmarks?
• RQ2 Do complex blocks (convolution, BN, activations) enable competitive performance with reasonable initialization and training stability?
• RQ3 Are complex networks particularly advantageous for audio-related tasks like music transcription and speech spectrum prediction?

Key findings

• Complex networks achieve competitive results with real-valued models on CIFAR-10, CIFAR-100, and SVHN*.
• On CIFAR-100, complex representations outperform real counterparts in the reported setup.
• A 2D whitening-based complex batch normalization avoids NaNs and stabilizes training across experiments.
• C-ReLU outperforms modReLU and z-ReLU in the reported image-recognition experiments.
• Ablations show the importance of complex batch normalization and phase-preserving activations for performance and stability.
• Experiments indicate state-of-the-art performance on MusicNet transcription and TIMIT spectrum prediction within their reported scope.