[Paper Review] Learning Neuron Non-Linearities with Kernel-Based Deep Neural Networks

Giuseppe Marra, Dario Zanca+2 more|arXiv (Cornell University)|Jul 17, 2018

Neural Networks and Applications11 references3 citations

TL;DR

This paper proposes Kernel-Based Deep Neural Networks (KBDNNs) that learn optimal neuron activation functions via kernel expansions in a regularized optimization framework, enabling superior performance on long-sequence tasks. Unlike standard ReLU or LSTM units, the learned non-linearities are non-monotonic and can alternate between contractive and expansive behaviors, effectively mitigating vanishing gradients and enabling KBRN to capture long-term dependencies with high accuracy even in sequences up to length 200.

ABSTRACT

The effectiveness of deep neural architectures has been widely supported in terms of both experimental and foundational principles. There is also clear evidence that the activation function (e.g. the rectifier and the LSTM units) plays a crucial role in the complexity of learning. Based on this remark, this paper discusses an optimal selection of the neuron non-linearity in a functional framework that is inspired from classic regularization arguments. It is shown that the best activation function is represented by a kernel expansion in the training set, that can be effectively approximated over an opportune set of points modeling 1-D clusters. The idea can be naturally extended to recurrent networks, where the expressiveness of kernel-based activation functions turns out to be a crucial ingredient to capture long-term dependencies. We give experimental evidence of this property by a set of challenging experiments, where we compare the results with neural architectures based on state of the art LSTM cells.

Motivation & Objective

• To address the limitation of fixed, monotonic activation functions in recurrent networks that hinder long-term dependency learning.
• To develop a unified framework where both network weights and neuron non-linearities are jointly optimized via regularization.
• To investigate whether kernel-based activation functions can outperform standard LSTM units in capturing long-range dependencies in sequential data.
• To provide theoretical and empirical evidence that non-monotonic, kernel-derived activation functions enable better gradient flow in deep recurrent architectures.

Proposed method

• Formulate the learning problem as a regularized optimization over both network weights and functional activation parameters, using a kernel expansion over training points.
• Represent each neuron's activation function as a kernel expansion: $ f(x) = \sum_{i=1}^{d} \alpha_i K(x, x_i) $, where $ x_i $ are centroids from the training set.
• Use a normalized kernel representation to ensure stability and generalization, with $ d=100 $ centroids used in experiments.
• Train the model end-to-end using Adam optimizer with $ \lambda = 0.001 $, updating both weights and kernel coefficients simultaneously.
• Extend the framework to recurrent networks by modeling the iterated map as a combination of contractive and expansive dynamics via non-monotonic activation functions.
• Design benchmarks where distinguishing information is only in the first few bits of long sequences, testing the model's ability to retain early information.

Experimental results

Research questions

• RQ1 Can kernel-based activation functions outperform standard fixed non-linearities like ReLU and sigmoid in capturing long-term dependencies?
• RQ2 Does the ability of kernel-based activation functions to be non-monotonic improve gradient flow and prevent vanishing gradients in recurrent networks?
• RQ3 To what extent can KBDNNs generalize to long sequences when the critical information is located only at the beginning?
• RQ4 How does the performance of KBRN compare to state-of-the-art LSTM-based architectures on sequence classification tasks with increasing sequence length?

Key findings

• KBRN achieved near-perfect success rates on all benchmarks with sequence lengths up to 150, including the equivalence (≡) function, while LSTMs began to fail beyond length 30.
• For the ≡ function, KBRN solved sequences of length 150 in all 5 runs, but failed on length 200, indicating a practical upper bound around 150 for this setup.
• KBRN converged significantly faster than LSTM networks, with average training iterations required decreasing as sequence length increased, unlike LSTMs which required more iterations and often failed to converge.
• The non-monotonic nature of the learned activation functions enabled both contractive and expansive behaviors during sequence processing, allowing effective backpropagation of long-range dependencies.
• On all four Boolean functions (OR, AND, XOR, EQUIVALENCE), KBRN achieved success in 90–100% of trials across all tested sequence lengths, while LSTMs dropped below 50% success rate for sequences longer than 30.
• The results demonstrate that kernel-based activation functions provide a more robust and expressive inductive bias than standard fixed non-linearities in recurrent architectures.