[Paper Review] Eluder Dimension and Generalized Rank.

Gene Li, Pritish Kamath+2 more|arXiv (Cornell University)|Apr 14, 2021

Machine Learning and Algorithms18 references4 citations

TL;DR

This paper investigates the relationship between eluder dimension and a generalized notion of rank tied to monotone activations $σ$, showing that eluder dimension can be exponentially smaller than $σ$-rank when $σ$ has derivatives bounded away from zero, but can also be exponentially larger when $σ$ is ReLU, demonstrating the necessity of the derivative condition.

ABSTRACT

We study the relationship between the eluder dimension for a function class and a generalized notion of rank, defined for any monotone activation $\sigma : \mathbb{R} o \mathbb{R}$, which corresponds to the minimal dimension required to represent the class as a generalized linear model. When $\sigma$ has derivatives bounded away from $0$, it is known that $\sigma$-rank gives rise to an upper bound on eluder dimension for any function class; we show however that eluder dimension can be exponentially smaller than $\sigma$-rank. We also show that the condition on the derivative is necessary; namely, when $\sigma$ is the $\mathrm{relu}$ activation, we show that eluder dimension can be exponentially larger than $\sigma$-rank.

Motivation & Objective

• To understand the relationship between eluder dimension and a generalized notion of rank for function classes under monotone activations.
• To determine whether the derivative condition on $σ$ is necessary for bounding eluder dimension via $σ$-rank.
• To analyze how the choice of activation function, particularly ReLU, affects the relative sizes of eluder dimension and $σ$-rank.
• To establish tight bounds on the gap between eluder dimension and $σ$-rank under different activation conditions.

Proposed method

• Define a generalized notion of rank, called $σ$-rank, for function classes based on the minimal dimension required to represent them as generalized linear models under a monotone activation $σ$.
• Establish that when $σ$ has derivatives bounded away from zero, $σ$-rank upper bounds eluder dimension.
• Construct explicit function classes where eluder dimension is exponentially smaller than $σ$-rank to demonstrate the tightness of the upper bound.
• Provide a construction showing that for ReLU activation, eluder dimension can be exponentially larger than $σ$-rank, proving the necessity of the derivative condition.
• Use structural properties of monotone activations and function class representations to derive the exponential separations.

Experimental results

Research questions

• RQ1 How does the eluder dimension of a function class relate to its $σ$-rank for a monotone activation $σ$?
• RQ2 Can the upper bound of eluder dimension by $σ$-rank be exponentially loose for activations with non-vanishing derivatives?
• RQ3 Is the condition that $σ'$ is bounded away from zero necessary for the $σ$-rank to upper bound eluder dimension?
• RQ4 What happens to the relationship between eluder dimension and $σ$-rank when $σ$ is ReLU, which has a derivative that vanishes on half the domain?

Key findings

• When $σ$ has derivatives bounded away from zero, $σ$-rank provides an upper bound on eluder dimension.
• There exist function classes for which eluder dimension is exponentially smaller than $σ$-rank, showing the upper bound is not always tight.
• For the ReLU activation, eluder dimension can be exponentially larger than $σ$-rank, demonstrating that the derivative condition is necessary for the upper bound.
• The exponential separation in both directions reveals a fundamental asymmetry in the relationship between eluder dimension and $σ$-rank depending on the activation function.