[Paper Review] Sobolev Training for Neural Networks

Wojciech Marian Czarnecki, Simon Osindero+3 more|arXiv (Cornell University)|Jun 15, 2017

Adversarial Robustness in Machine LearningComputer Science18 references60 citations

TL;DR

The paper introduces Sobolev Training to incorporate target derivatives into neural network training, improving data efficiency and generalization across regression, distillation, and synthetic gradient contexts.

ABSTRACT

At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.

Motivation & Objective

• Introduce Sobolev Training as a paradigm for neural networks.
• Theoretically justify that matching derivatives alongside values is feasible and beneficial.
• Provide empirical evidence across regression benchmarks, policy distillation, and synthetic gradient scenarios.

Proposed method

• Extend standard loss to include derivative matching terms for first (and higher) order derivatives (Sobolev loss).
• Provide a stochastic variant that projects derivatives onto random vectors to reduce computational cost.
• Theoretically establish universal approximation in Sobolev spaces for ReLU networks and discuss sample complexity benefits.
• Demonstrate applications in three domains: regression on optimisation benchmarks, Atari policy distillation, and large-scale synthetic gradient training.

Experimental results

Research questions

• RQ1 Can Sobolev Training reduce sample complexity compared to standard value-only training?
• RQ2 Do neural networks with Sobolev Training accurately approximate both function values and derivatives in practice?
• RQ3 How does Sobolev Training affect performance in distillation and synthetic gradient contexts?
• RQ4 What are the theoretical guarantees for approximation in Sobolev spaces with modern activations like ReLU?

Key findings

• Sobolev Training yields lower approximation error and better generalization in low-data regimes across benchmark functions.
• In distillation tasks, Sobolev Training produces distilled policies more similar to target policies (lower test KL divergence).
• For synthetic gradients, Sobolev Training improves final accuracy and stability compared to baseline gradient methods, including ImageNet-scale experiments.
• The paper provides theoretical results showing ReLU networks can universal-approximate in Sobolev spaces (K=1) and discusses sample-complexity advantages.
• Empirical results span regression, Atari policy distillation, and large-scale synthetic gradient applications, with consistent improvements over baselines.