[Paper Review] Learning to Teach with Dynamic Loss Functions

Lijun Wu, Fei Tian+5 more|arXiv (Cornell University)|Oct 29, 2018

Machine Learning and Algorithms42 references42 citations

TL;DR

The paper introduces L2T-DLF, a framework where a neural teacher outputs dynamic loss functions to guide a student model during training, optimized via gradient-based reverse-mode differentiation, improving performance in image classification and neural machine translation.

ABSTRACT

Teaching is critical to human society: it is with teaching that prospective students are educated and human civilization can be inherited and advanced. A good teacher not only provides his/her students with qualified teaching materials (e.g., textbooks), but also sets up appropriate learning objectives (e.g., course projects and exams) considering different situations of a student. When it comes to artificial intelligence, treating machine learning models as students, the loss functions that are optimized act as perfect counterparts of the learning objective set by the teacher. In this work, we explore the possibility of imitating human teaching behaviors by dynamically and automatically outputting appropriate loss functions to train machine learning models. Different from typical learning settings in which the loss function of a machine learning model is predefined and fixed, in our framework, the loss function of a machine learning model (we call it student) is defined by another machine learning model (we call it teacher). The ultimate goal of teacher model is cultivating the student to have better performance measured on development dataset. Towards that end, similar to human teaching, the teacher, a parametric model, dynamically outputs different loss functions that will be used and optimized by its student model at different training stages. We develop an efficient learning method for the teacher model that makes gradient based optimization possible, exempt of the ineffective solutions such as policy optimization. We name our method as "learning to teach with dynamic loss functions" (L2T-DLF for short). Extensive experiments on real world tasks including image classification and neural machine translation demonstrate that our method significantly improves the quality of various student models.

Motivation & Objective

• Motivate and formalize the concept of loss function teaching in AI, drawing an analogy to human teaching and exams.
• Develop a gradient-based optimization framework to jointly train a teacher (loss-function generator) and a student (learner).
• Demonstrate that dynamically learned loss functions improve student performance on real-world tasks.
• Provide an efficient algorithm that can backpropagate through the training process using reverse-mode differentiation.

Proposed method

• Define a student model f_ω and a learnable loss l_Φ that guides training via SGD.
• Introduce a teacher model μ_θ that outputs loss-function coefficients Φ_t based on the student’s status s_t, enabling dynamic loss functions during training.
• Relax non-differentiable task-specific metrics m with a continuous surrogate using randomness from the student’s output p_ω to obtain a differentiable objective.
• Apply Reverse-Mode Differentiation (RMD) to backpropagate through the entire training process and derive gradients dθ for the teacher parameters.
• Use gradient-based optimization (e.g., Adam) to update the teacher, iterating to maximize development-set performance of the resulting student.
• Illustrate concrete instantiations in image classification and neural machine translation (NMT), including loss forms like l_Φ(p, y) = -σ(y^T Φ log p) and attention-based Φ_t outputs.

Experimental results

Research questions

• RQ1 Can a neural teacher learn to output loss functions that improve a student’s development-set performance over fixed losses?
• RQ2 How can we efficiently optimize the teacher to adapt loss functions to different training phases of the student?
• RQ3 Do dynamically learned loss functions generalize across tasks such as image classification and neural machine translation?
• RQ4 What insights can be gained about the structure of learned loss functions during training?

Key findings

• Dynamic loss functions learned by the teacher yield improvements across multiple student architectures and tasks.
• On CIFAR-10 with various models, the teacher-enhanced losses achieve lower error rates, e.g., WRN reaches 3.42% on CIFAR-10 and DenseNet-BC improves to 3.08% on CIFAR-10.
• On MNIST, L2T-DLF-enabled training yields lower error rates across models like MLP, LeNet, and beyond.
• In NMT tasks (IWSLT-14 German→English), L2T-DLF improves BLEU scores across LSTM-1, LSTM-2, and Transformer students (e.g., Transformer from 34.01 to 34.80 BLEU).
• The learned loss coefficients Φ_t reveal phase-dependent focusing effects, such as promoting similarity between easy classes early and tightening discrimination between similar classes later.