[Paper Review] Continuously Differentiable Exponential Linear Units

Jonathan T. Barron|arXiv (Cornell University)|Apr 24, 2017

Control Systems and Identification1 references103 citations

TL;DR

The paper introduces CELU, a continuously differentiable variant of ELU with a unified parameterization that ensures smoothness, bounded derivatives, scale-similarity, and contains both ReLU and linear functions as special cases. It also provides explicit derivatives w.r.t. x and alpha and shows limiting behavior as alpha varies.

ABSTRACT

Exponential Linear Units (ELUs) are a useful rectifier for constructing deep learning architectures, as they may speed up and otherwise improve learning by virtue of not have vanishing gradients and by having mean activations near zero. However, the ELU activation as parametrized in [1] is not continuously differentiable with respect to its input when the shape parameter alpha is not equal to 1. We present an alternative parametrization which is C1 continuous for all values of alpha, making the rectifier easier to reason about and making alpha easier to tune. This alternative parametrization has several other useful properties that the original parametrization of ELU does not: 1) its derivative with respect to x is bounded, 2) it contains both the linear transfer function and ReLU as special cases, and 3) it is scale-similar with respect to alpha.

Motivation & Objective

• Address the discontinuity in ELU derivatives when alpha != 1.
• Propose a new parametrization that is C1 continuous for all alpha values.
• Ensure the derivative with respect to x is bounded and the model remains easy to optimize.
• Show that CELU includes linear and ReLU as special cases and is scale-similar in alpha.

Proposed method

• Define CELU as CELU(x, alpha) = x for x >= 0 and alpha( exp(x/alpha) - 1 ) for x < 0.
• Derive the derivative with respect to x: d/dx CELU(x, alpha) = 1 if x >= 0, exp(x/alpha) otherwise.
• Derive the derivative with respect to alpha: d/d alpha CELU(x, alpha) = 0 if x >= 0, exp(x/alpha)(1 - x/alpha) - 1 otherwise.
• Show scale-similarity: CELU(x, alpha) = (1/c) CELU(c x, c alpha).
• Highlight that derivative with respect to x is bounded and efficient computation uses exp(x/alpha).
• Explain limiting cases: as alpha -> 0+, CELU -> ReLU; as alpha -> infinity, CELU -> x; and ELU with alpha=1 matches CELU.

Experimental results

Research questions

• RQ1 Does the CELU parametrization remain continuously differentiable for all alpha values?
• RQ2 What are the derivative properties of CELU with respect to x and alpha?
• RQ3 How does CELU relate to and interpolate between ReLU and linear activations?
• RQ4 What practical benefits does CELU provide compared to the original ELU in training dynamics?

Key findings

• CELU is C1 continuous for all alpha values.
• The derivative with respect to x is bounded for CELU.
• CELU contains both linear transfer and ReLU as special cases.
• CELU is scale-similar: CELU(x, alpha) = (1/c) CELU(c x, c alpha).
• As alpha -> 0+, CELU converges to ReLU; as alpha -> infinity, CELU converges to the linear function.
• CELU equals ELU when alpha = 1.