[Paper Review] Teaching with Commentaries

Aniruddh Raghu, Maithra Raghu+3 more|arXiv (Cornell University)|Nov 5, 2020

Domain Adaptation and Few-Shot Learning37 references4 citations

TL;DR

This paper introduces a flexible learning-to-teach framework using 'commentaries'—learned meta-information that improves neural network training. By leveraging gradient-based optimization via implicit differentiation, commentaries enable faster convergence, better performance, and improved robustness across tasks such as example weighting, data augmentation, and attention masking, with reusable commentaries generalizing across models and datasets.

ABSTRACT

Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural network training examine teaching: providing learned information during the training process to improve downstream model performance. In this paper, we take steps towards extending the scope of teaching. We propose a flexible teaching framework using commentaries, learned meta-information helpful for training on a particular task. We present gradient-based methods to learn commentaries, leveraging recent work on implicit differentiation for scalability. We explore diverse applications of commentaries, from weighting training examples, to parameterising label-dependent data augmentation policies, to representing attention masks that highlight salient image regions. We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process. We also observe that commentaries generalise: they can be reused when training new models to obtain performance benefits, suggesting a use-case where commentaries are stored with a dataset and leveraged in future for improved model training.

Motivation & Objective

• Address the challenge of making deep neural network training faster and more data-efficient by learning auxiliary teaching signals.
• Overcome limitations of prior teaching methods that are task-specific and not reusable across models or datasets.
• Develop a unified, flexible framework for learning meta-information (commentaries) that can guide training and provide interpretability.
• Demonstrate that commentaries generalize across models, enabling reuse with new training setups to improve performance and speed.
• Explore diverse applications of commentaries beyond accuracy—such as interpretability (attention masks) and robustness to spurious correlations.

Proposed method

• Define commentaries as learned meta-information (e.g., example weights, augmentation policies, attention masks) parameterized by a neural network or tabular function.
• Use backpropagation through training to optimize commentary parameters by differentiating the student network’s validation loss with respect to commentary parameters.
• Leverage implicit differentiation and hypergradient computation (via Lorraine et al., 2020) to scale commentary learning to large models.
• Apply commentaries in three distinct ways: (1) curriculum learning via learned example weights, (2) label-dependent data augmentation policies, and (3) attention mask generation for saliency detection.
• Train commentary networks end-to-end using a validation loss objective, enabling the model to learn meta-knowledge useful for downstream training.
• Store and reuse commentaries with datasets to improve training of new models, demonstrating transferability and generalization.

Experimental results

Research questions

• RQ1 Can learned meta-information (commentaries) improve training speed and performance across diverse deep learning tasks?
• RQ2 Can commentaries provide interpretable insights into data and training dynamics, such as identifying salient image regions?
• RQ3 Do commentaries generalize across models and datasets, enabling reuse to accelerate or improve new training runs?
• RQ4 How effective are commentaries in mitigating spurious correlations, such as background features in the CUB dataset?
• RQ5 Can commentaries be used to learn effective data augmentation policies that improve model robustness and accuracy?

Key findings

• Commentaries trained via backpropagation through training significantly improve validation accuracy and reduce training time on few-shot learning benchmarks.
• Learned attention masks on the CUB-200-2011 dataset focus on the bird rather than the background, demonstrating interpretability and robustness.
• On the CUB dataset with distribution shift (spurious background correlations), using attention masks improved test set top-1 accuracy from 25.81% to 30.46% and top-5 accuracy from 56.98% to 61.57%.
• Commentaries for data augmentation led to improved performance on benchmark tasks compared to baseline augmentation strategies.
• Example-weighting curricula learned via commentaries provided interpretable insights into which training examples are most informative.
• Commentaries generalize: reusing learned commentaries with new models (e.g., ResNet-18) leads to measurable improvements in training speed and performance, suggesting a viable pipeline for dataset-level knowledge reuse.