[Paper Review] SGD Learns the Conjugate Kernel Class of the Network

Amit Daniely|arXiv (Cornell University)|Feb 27, 2017

Stochastic Gradient Optimization Techniques81 citations

TL;DR

Demonstrates that SGD can learn, in polynomial time, functions in the conjugate kernel space of a broad family of deep networks, by training the full network and relating to kernel dynamics after random initialization.

ABSTRACT

We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely, Frostig and Singer. The result holds for log-depth networks from a rich family of architectures. To the best of our knowledge, it is the first polynomial-time guarantee for the standard neural network learning algorithm for networks of depth more that two. As corollaries, it follows that for neural networks of any depth between $2$ and $\log(n)$, SGD is guaranteed to learn, in polynomial time, constant degree polynomials with polynomially bounded coefficients. Likewise, it follows that SGD on large enough networks can learn any continuous function (not in polynomial time), complementing classical expressivity results.

Motivation & Objective

• Motivate and formalize when standard SGD succeeds for neural networks beyond depth-2 architectures.
• Link neural network training to a conjugate kernel framework via random initialization.
• Prove polynomial-time guarantees for SGD to learn functions in the kernel space of the network under broad conditions.
• Show corollaries: efficient learning of constant-degree polynomials and learning of any continuous function with sufficient network size.

Proposed method

• Model neural networks as computation skeletons with replication and output layers.
• Define conjugate activations and compositional kernels associated with skeletons.
• Analyze SGD with Xavier-like random initialization and zero-initialized prediction layer.
• Prove that with polynomial network size, learning rate, and steps, SGD achieves near-optimal loss in the kernel space.
• Provide corollaries translating kernel-space results to practical function classes (polynomials, CNF/DNF, continuous functions).

Experimental results

Research questions

• RQ1 Can SGD on deep architectures guarantee learning any function in the network's conjugate kernel space within polynomial time?
• RQ2 How do network depth, replication, and activation bounds affect the convergence guarantees of SGD to kernel-space targets?
• RQ3 What classes of functions are efficiently learnable by SGD via the conjugate kernel framework (e.g., polynomials, logical formulas, continuous functions)?

Key findings

• SGD is guaranteed to learn, in polynomial time, constant-degree polynomials with polynomially bounded coefficients in the conjugate kernel space.
• SGD is guaranteed to learn, in polynomial time, conjunctions, DNF/CNF with certain constants terms/literals within the kernel class.
• For networks with depth up to log(n), SGD learns functions in the conjugate kernel space, including non-polynomial classes like continuous functions (not necessarily in polynomial time).
• There exist corollaries showing SGD can learn continuous functions and certain PAC-learnable function classes within the kernel framework.
• The results extend to fully connected and convolutional networks with logarithmic-depth and constants in the number of convolutional layers.