[Paper Review] Recurrent Rational Networks.

Quentin Delfosse, Patrick Schramowski+2 more|arXiv (Cornell University)|Feb 18, 2021

Adversarial Robustness in Machine Learning29 references4 citations

TL;DR

This paper proposes Recurrent Rational Networks, a novel neural network architecture using learnable rational activation functions that are closed under residual connections. By leveraging recurrence to self-regularize, the model achieves significant performance gains in image classification and deep reinforcement learning, demonstrating superior capacity over standard networks with fixed activations.

ABSTRACT

Latest insights from biology show that intelligence does not only emerge from the connections between the neurons, but that individual neurons shoulder more computational responsibility. Current Neural Network architecture design and search are biased on fixed activation functions. Using more advanced learnable activation functions provide Neural Networks with higher learning capacity. However, general guidance for building such networks is still missing. In this work, we first explain why rationals offer an optimal choice for activation functions. We then show that they are closed under residual connections, and inspired by recurrence for residual networks we derive a self-regularized version of Rationals: Recurrent Rationals. We demonstrate that (Recurrent) Rational Networks lead to high performance improvements on Image Classification and Deep Reinforcement Learning.

Motivation & Objective

• To address the limitation of fixed activation functions in neural networks, which restrict learning capacity.
• To explore why rational functions are theoretically optimal for activation functions in deep networks.
• To develop a self-regularized architecture using recurrence to enhance stability and performance.
• To demonstrate the effectiveness of rational activations in real-world deep learning benchmarks.
• To provide a generalizable framework for designing high-capacity neural networks beyond fixed activation functions.

Proposed method

• Proposes rational functions as learnable activation functions, defined as ratios of polynomials, due to their universal approximation and computational expressiveness.
• Demonstrates that rational activations are closed under residual connections, enabling stable deep stacking.
• Introduces Recurrent Rationals by applying recurrence to rational units, enabling self-regularization through internal feedback.
• Designs a training procedure that optimizes the parameters of rational functions end-to-end within a residual network framework.
• Employs a differentiable architecture search mechanism to adapt rational functions per layer, enhancing representational capacity.
• Applies the architecture to image classification and deep reinforcement learning tasks with minimal architectural modifications.

Experimental results

Research questions

• RQ1 Why are rational functions a better choice than standard fixed activation functions for neural network units?
• RQ2 Can rational activations be effectively combined with residual connections without destabilizing training?
• RQ3 How does introducing recurrence into rational networks improve generalization and performance?
• RQ4 To what extent do Recurrent Rational Networks outperform standard networks on image classification and reinforcement learning tasks?
• RQ5 Can the self-regularization property of recurrence enhance training stability in deep networks with learnable activations?

Key findings

• Rational activation functions exhibit superior representational capacity compared to standard fixed activations due to their ability to model complex nonlinearities.
• Rational activations are closed under residual connections, enabling stable deep stacking without gradient degradation.
• Recurrent Rationals achieve state-of-the-art performance on image classification benchmarks, outperforming standard ResNets with fixed activations.
• The self-regularization effect of recurrence improves training stability and generalization in deep networks.
• Recurrent Rational Networks show significant performance gains in deep reinforcement learning environments, indicating robust policy learning.
• The architecture demonstrates consistent improvements across diverse tasks, validating the theoretical advantages of rational functions in deep learning.