[Paper Review] A Survey of Complex-Valued Neural Networks

Joshua Bassey, Lijun Qian+1 more|arXiv (Cornell University)|Jan 28, 2021

Neural Networks and ApplicationsComputer Science148 references78 citations

TL;DR

This paper provides a systematic survey of complex-valued neural networks (CVNNs), categorizing their activation functions, learning/optimization methods, input/output representations, and applications, and discusses challenges and future directions.

ABSTRACT

Artificial neural networks (ANNs) based machine learning models and especially deep learning models have been widely applied in computer vision, signal processing, wireless communications, and many other domains, where complex numbers occur either naturally or by design. However, most of the current implementations of ANNs and machine learning frameworks are using real numbers rather than complex numbers. There are growing interests in building ANNs using complex numbers, and exploring the potential advantages of the so-called complex-valued neural networks (CVNNs) over their real-valued counterparts. In this paper, we discuss the recent development of CVNNs by performing a survey of the works on CVNNs in the literature. Specifically, a detailed review of various CVNNs in terms of activation function, learning and optimization, input and output representations, and their applications in tasks such as signal processing and computer vision are provided, followed by a discussion on some pertinent challenges and future research directions.

Motivation & Objective

• Provide a systematic review and categorization of state-of-the-art CVNNs by activation, learning/optimization, input/output representations, and applications.
• Compare and contrast different schools of thought, highlighting similarities, differences, advantages, and limitations of CVNN approaches.
• Discuss challenges facing CVNNs and outline promising directions for future research.
• Establish a foundation for researchers to understand when and why to use CVNNs in signal processing, communications, and vision.

Proposed method

• Survey and categorization of CVNN literature along activation function types, learning/optimization approaches, and input/output representations.
• Discussion of activation-function properties, including holomorphic vs non-holomorphic and split activations (Type A vs Type B).
• Explanation of learning methods, including gradient-based backpropagation with Wirtinger calculus and non-gradient derivative-free approaches.
• Presentation of mathematical formulations for complex loss (L), log-loss, and backpropagation updates in the complex domain.
• Overview of historical context and rationale for complex-valued representations in CVNNs.

Experimental results

Research questions

• RQ1 What activation functions have been explored for CVNNs and how do they differ (holomorphic vs non-holomorphic, split-type, ETFs, ReLU adaptations)?
• RQ2 What learning and optimization strategies are used for CVNNs (gradient-based with Wirtinger calculus, non-gradient methods, second-order methods)?
• RQ3 How are inputs and outputs represented in CVNNs (split-real, split-phase, fully complex, MVN/MLMVN frameworks)?
• RQ4 In what applications do CVNNs show potential advantages, and what challenges hinder their broader adoption?

Key findings

• CVNNs employ a wide spectrum of activation functions, including split-type, fully complex, ETF-based, non-parametric, and ReLU-like variants, with no consensus on a single best choice.
• Wirtinger calculus is a foundational tool for deriving complex gradients in gradient-based CVNN training, enabling complex-domain backpropagation.
• There are distinct learning paradigms in CVNNs, including gradient-based backpropagation and derivative-free, error-correction approaches for multi-valued neurons.
• Input/output representations in CVNNs vary from split-real and split-amplitude/phase to fully complex, with multi-valued neurons offering unit-circle outputs.
• The survey emphasizes the trade-offs between holomorphic constraints and practical non-holomorphic activations, highlighting challenges in regularization, initialization, and stability.
• The work consolidates historical developments and outlines future directions, suggesting CVNNs’ potential in domains where complex-valued data are natural (e.g., signal processing, communications).