[Paper Review] Neural Networks Fail to Learn Periodic Functions and How to Fix It

Liu Ziyin, Tilman Hartwig+1 more|arXiv (Cornell University)|Jun 15, 2020

Neural Networks and ApplicationsComputer Science38 references58 citations

TL;DR

The paper shows standard activations fail to extrapolate periodic functions and introduces Snake activation (x + sin^2(x)) to bias toward periodicity, with universal extrapolation theorem and real-world tests.

ABSTRACT

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

Motivation & Objective

• Assess how common activation functions extrapolate beyond bounded training regions for periodic functions.
• Demonstrate the inadequacy of ReLU, tanh, and their variants in learning periodicity.
• Propose a periodic-inductive-bias activation (Snake) and analyze its optimization and initialization.
• Prove an extrapolation theorem showing Snake networks can universally approximate well-behaved periodic functions.
• Validate Snake on synthetic, climate/temperature and financial time-series data.

Proposed method

• The authors analyze extrapolation properties of networks with ReLU, tanh, and related activations both experimentally and theoretically.
• They prove two extrapolation theorems showing asymptotic linear or constant behavior for ReLU/tanh networks.
• They introduce Snake activation: Snake_a(x) = x + (1/a) sin^2(a x) and discuss monotonicity and optimization advantages.
• They compare variants like sin, x+sin(x), and x+sin^2(x) to establish advantages of Snake, including a frequency parameter a.
• They derive initialization considerations for Snake to maintain unit pre-activation variance across layers.
• They prove a universal extrapolation theorem: a wide Snake network with sufficient width can uniformly converge to any piecewise C^1 periodic function with period L.
• They apply Snake in image classification (CIFAR-10), atmospheric temperature prediction, body temperature, and financial data (Wilshire 5000), and compare against baselines.

Experimental results

Research questions

• RQ1 Can standard activation functions extrapolate periodic patterns beyond the training interval?
• RQ2 Does introducing a periodic inductive bias via a new activation enable learning and extrapolation of periodic functions?
• RQ3 How does Snake perform on real-world periodic or quasi-periodic time series compared with traditional activations?
• RQ4 What initialization and architectural considerations maximize Snake performance?
• RQ5 Is Snake capable of universal extrapolation for well-behaved periodic functions?

Key findings

• Standard activations (ReLU, tanh, Swish, sin-based) fail to extrapolate simple periodic functions beyond the training region.
• Snake activation x + sin^2(x) induces a periodic inductive bias and enables accurate interpolation and extrapolation of periodic signals.
• Snake achieves competitive performance on CIFAR-10 as a general-purpose activation and excels on temperature and financial time-series tasks compared with common baselines.
• An explicit universal extrapolation theorem shows Snake networks with sufficient width can approximate any piecewise C^1 periodic function uniformly (continuous case).
• Proper initialization for Snake (variance scaling) improves training speed and convergence, with practical default a ≈ 0.5 for general tasks and higher a for explicit periodicity.