[Paper Review] Neural tangent kernels, transportation mappings, and universal approximation

Ziwei Ji, Matus Telgarsky+1 more|arXiv (Cornell University)|Apr 30, 2020

Stochastic Gradient Optimization Techniques29 references12 citations

TL;DR

This paper establishes universal approximation rates for the neural tangent kernel (NTK) by modeling the target function as an infinite-width linearized network with weights derived from a transport mapping of random initializations. It shows that with high probability, perturbing initial weights by O(B/(eps * sqrt(m))) yields an NTK approximation within (sqrt(eps) + B/sqrt(eps * m))^2 of both the target function and the corresponding shallow network, matching known shallow network rates and implying universal approximation is intrinsic to the NTK regime.

ABSTRACT

This paper establishes rates of universal approximation for the neural tangent kernel (NTK) in the standard setting of microscopic changes to initial weights. Concretely, given a target function f, a target width m, and a target approximation error eps>0, then with high probability, moving the initial weight vectors a distance B/(eps * sqrt{m}) will give a linearized finite-width NTK which is (sqrt(eps) + B/sqrt(eps * m))^2-close to both the target function f, and also the shallow network which this NTK linearized. The constant B can be independent of eps --- particular cases studied here include f having good Fourier transform or RKHS norm --- though in the worse case it scales roughly as 1/eps^d for general continuous functions. The method of proof is to rewrite f with equality as an infinite-width linearized network whose weights are a transport mapping applied to random initialization, and to then sample from this transport mapping. This proof therefore provides another perspective on the scaling behavior of the NTK: redundancy in the weights due to resampling allows weights to be scaled down. Since the approximation rates match those in the literature for shallow networks, this work implies that universal approximation is not reliant upon any behavior outside the NTK regime.

Motivation & Objective

• To establish quantitative rates of universal approximation for the neural tangent kernel (NTK) in the finite-width, linearized regime.
• To understand the dependence of approximation error on target function complexity, network width m, and desired accuracy eps.
• To show that universal approximation in the NTK regime does not require dynamics beyond the linearized kernel, matching shallow network performance.
• To provide a new perspective on NTK scaling by interpreting weight redundancy through resampling and transport mappings.

Proposed method

• Rewriting the target function f as an infinite-width linearized network whose weights are generated by applying a transport mapping to random initial weight vectors.
• Defining the transport mapping such that the resulting infinite-width network exactly represents f, enabling exact decomposition of the target function.
• Sampling from the transport mapping to construct finite-width approximations of the NTK, ensuring high-probability approximation guarantees.
• Bounding the approximation error between the finite-width NTK and both f and the corresponding shallow network using the expression (sqrt(eps) + B/sqrt(eps * m))^2.
• Analyzing the constant B in terms of function properties such as Fourier transform behavior or RKHS norm, showing it can be independent of eps in favorable cases.
• Demonstrating that the worst-case scaling of B is roughly 1/eps^d for general continuous functions, but better in structured settings.

Experimental results

Research questions

• RQ1 What is the quantitative rate at which the neural tangent kernel can approximate a given target function f in the finite-width, linearized regime?
• RQ2 How does the required perturbation of initial weights scale with network width m and desired accuracy eps to achieve universal approximation?
• RQ3 Can the approximation error in the NTK regime match known rates for shallow networks, and if so, under what conditions?
• RQ4 What role does the transport mapping of initial weights play in enabling high-probability universal approximation?
• RQ5 Is universal approximation in the NTK regime dependent on nonlinear dynamics outside the linearized kernel, or is it intrinsic to the NTK itself?

Key findings

• The finite-width NTK achieves an approximation error of (sqrt(eps) + B/sqrt(eps * m))^2 to both the target function f and the corresponding shallow network with high probability.
• The constant B can be independent of eps when f has favorable properties such as good Fourier transform or bounded RKHS norm.
• In the worst case, B scales as roughly 1/eps^d for general continuous functions, reflecting the complexity of the target function.
• The method provides a new interpretation of NTK scaling: redundancy from resampling allows weights to be scaled down without loss of representational capacity.
• The approximation rates match those known for shallow networks, indicating that universal approximation in the NTK regime does not rely on nonlinear behavior beyond the linearized kernel.
• The work confirms that universal approximation is inherent to the NTK framework, without requiring dynamics outside the linearized regime.