[Paper Review] Limitations of the Empirical Fisher Approximation.

Frederik Künstner, Lukas Balles+1 more|arXiv (Cornell University)|May 29, 2019

Stochastic Gradient Optimization TechniquesComputer Science18 references18 citations

TL;DR

This paper challenges the use of the empirical Fisher as a proxy for the true Fisher information matrix in natural gradient methods. It demonstrates that the empirical Fisher fails to capture second-order information in practice, leading to suboptimal optimization behavior even on simple problems, and shows that the conditions under which it approximates the Fisher or Hessian are rarely met.

ABSTRACT

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-order methods and heuristics like Adam. We dispute this argument by showing that the empirical Fisher---unlike the Fisher---does not generally capture second-order information. We further argue that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects.

Motivation & Objective

• To challenge the widespread assumption that the empirical Fisher approximates the true Fisher information matrix in natural gradient descent.
• To investigate whether the empirical Fisher can reliably capture second-order optimization information, as claimed by several influential works.
• To identify the practical limitations and pathologies of using the empirical Fisher in optimization algorithms like Adam.
• To clarify the conditions under which the empirical Fisher might approximate the Fisher or Hessian, and assess their feasibility in real-world settings.

Proposed method

• Analyzing the mathematical relationship between the empirical Fisher, the true Fisher information matrix, and the Hessian of the loss.
• Deriving conditions under which the empirical Fisher converges to the true Fisher or Hessian, and evaluating their practical feasibility.
• Conducting empirical evaluations on simple optimization problems to demonstrate the pathological behavior of the empirical Fisher.
• Contrasting the behavior of natural gradient descent using the true Fisher with that using the empirical Fisher to highlight performance differences.

Experimental results

Research questions

• RQ1 Under what conditions does the empirical Fisher approximate the true Fisher information matrix?
• RQ2 Does the empirical Fisher capture meaningful second-order information for optimization, as claimed by prior works?
• RQ3 How do the optimization dynamics differ when using the empirical Fisher versus the true Fisher in natural gradient descent?
• RQ4 Are the assumptions required for the empirical Fisher to approximate the Hessian or Fisher realistic in practical deep learning settings?

Key findings

• The empirical Fisher does not generally capture second-order information, unlike the true Fisher information matrix.
• The conditions under which the empirical Fisher approaches the true Fisher or Hessian are highly unlikely to be met in practice.
• Even on simple optimization problems, the empirical Fisher exhibits pathological behavior that degrades optimization performance.
• The use of the empirical Fisher in methods like Adam is not justified by second-order information capture, undermining the theoretical basis of such heuristics.