[Paper Review] Learning the Non-linearity in Convolutional Neural Networks

Gavneet Singh Chadha, Andreas Schwung|arXiv (Cornell University)|May 29, 2019

Neural Networks and Applications23 references4 citations

TL;DR

This paper proposes Nonlinear Convolutional Neural Networks (NLCNNs) that enhance feature learning by introducing trainable exponential nonlinearities via an exponent weight matrix (EWM) applied to receptive fields. The method improves classification performance on time series data—especially in nonlinear systems like the Tennessee Eastman Process—by increasing representational capacity and reducing optimization difficulties, with end-to-end training of EWM yielding superior results over fixed or data-augmentation-only approaches.

ABSTRACT

We propose the introduction of nonlinear operation into the feature generation process in convolutional neural networks. This nonlinearity can be implemented in various ways. First we discuss the use of nonlinearities in the process of data augmentation to increase the robustness of the neural networks recognition capacity. To this end, we randomly disturb the input data set by applying exponents within a certain numerical range to individual data points of the input space. Second we propose nonlinear convolutional neural networks where we apply the exponential operation to each element of the receptive field. To this end, we define an additional weight matrix of the same dimension as the standard kernel weight matrix. The weights of this matrix then constitute the exponents of the corresponding components of the receptive field. In the basic setting, we keep the weight parameters fixed during training by defining suitable parameters. Alternatively, we make the exponential weight parameters end-to-end trainable using a suitable parameterization. The network architecture is applied to time series analysis data set showing a considerable increase in the classification performance compared to baseline networks.

Motivation & Objective

• To address the limited representational capacity of standard CNNs in modeling inherently nonlinear processes, especially in time series analysis.
• To enhance robustness and generalization by introducing nonlinear data augmentation through random exponentiation of input data points.
• To propose a novel CNN architecture with an end-to-end trainable exponent weight matrix (EWM) that applies nonlinear transformations directly within the convolution operation.
• To evaluate the effectiveness of nonlinear operations in improving classification accuracy on real-world, nonlinear time series datasets such as the Tennessee Eastman Process.

Proposed method

• Introduces nonlinear data augmentation by randomly assigning exponents from a uniform range [-2, 4] to input data points, enhancing robustness during training.
• Proposes a nonlinear convolution operation where each element of the receptive field is raised to a learnable exponent defined by an additional weight matrix (EWM) of the same size as the kernel.
• The EWM is shared across all neurons in a feature map and can be initialized with meaningful patterns or trained end-to-end alongside standard convolutional weights.
• Employs a parameterization that allows the EWM to be differentiable and trainable, enabling the network to learn optimal nonlinearities during backpropagation.
• Uses a standard CNN architecture with modified convolution layers that apply element-wise exponentiation before linear combination, preserving computational efficiency.
• Applies the method to time series classification using the Tennessee Eastman Process benchmark, with data normalized to zero mean and unit covariance.

Experimental results

Research questions

• RQ1 Can nonlinear data augmentation via random exponentiation improve the generalization and robustness of CNNs in time series classification tasks?
• RQ2 Does incorporating learnable exponential nonlinearities directly into the convolution operation enhance the representational capacity of CNNs for nonlinear systems?
• RQ3 How does end-to-end training of the exponent weight matrix (EWM) compare to fixed or randomly sampled EWMs in terms of classification performance?
• RQ4 To what extent does the proposed nonlinear CNN architecture reduce optimization challenges such as local minima in training?
• RQ5 Can the NLCNN framework be effectively applied to real-world, complex, nonlinear industrial processes like the Tennessee Eastman Process?

Key findings

• Nonlinear data augmentation using random exponentiation improves classification performance over standard CNNs, though suboptimal exponent choices can degrade results.
• The introduction of a fixed EWM matrix significantly enhances model capacity by enabling nonlinear feature transformations within the convolution layer.
• End-to-end training of the EWM leads to a notable increase in classification accuracy compared to both fixed EWM and baseline CNNs.
• The proposed NLCNN architecture demonstrates improved robustness and generalization on the Tennessee Eastman Process benchmark, a complex, nonlinear industrial time series dataset.
• The method introduces minimal additional parameters, enabling efficient training and better generalization compared to other nonlinear kernel-based approaches.
• Preliminary results indicate that nonlinear operations flatten the loss landscape, reducing susceptibility to local minima during optimization.