[Paper Review] Everything old is new again: A multi-view learning approach to learning using privileged information and distillation

Weiran Wang|arXiv (Cornell University)|Mar 8, 2019

Advanced Bandit Algorithms Research16 references3 citations

TL;DR

This paper proposes a unified multi-view learning framework that integrates learning using privileged information (LUPI) and knowledge distillation by encouraging agreement between student and teacher predictors through a regularized empirical risk minimization (RERM) objective. Under optimistic assumptions about model performance, the method achieves faster convergence rates—specifically, an improved O(1/n) rate—by reducing the effective hypothesis space via canonical correlation-based regularization.

ABSTRACT

We adopt a multi-view approach for analyzing two knowledge transfer settings---learning using privileged information (LUPI) and distillation---in a common framework. Under reasonable assumptions about the complexities of hypothesis spaces, and being optimistic about the expected loss achievable by the student (in distillation) and a transformed teacher predictor (in LUPI), we show that encouraging agreement between the teacher and the student leads to reduced search space. As a result, improved convergence rate can be obtained with regularized empirical risk minimization.

Motivation & Objective

• To unify the analysis of learning using privileged information (LUPI) and knowledge distillation under a common multi-view learning framework.
• To investigate how agreement between teacher and student predictors reduces the search space for optimal models.
• To establish theoretical conditions under which faster convergence rates (O(1/n)) are achievable in both LUPI and distillation settings.
• To formalize the role of canonical correlation analysis (CCA) in controlling model complexity and improving generalization.
• To provide a practical, optimizable regularizer based on prediction discrepancy that enables improved generalization in both settings.

Proposed method

• Treat the standard features and privileged information (in LUPI) or soft targets (in distillation) as two views of the input data.
• Formulate a regularized empirical risk minimization (RERM) objective that penalizes the squared difference between student and teacher predictions.
• Use canonical correlation analysis (CCA) to define a coordinate system that controls the complexity of the hypothesis class.
• Assume optimistic performance: the student can achieve low expected loss in distillation, and a transformed teacher predictor performs well in LUPI.
• Derive convergence bounds showing that regularization leads to a larger range of optimal loss values achieving O(1/n) convergence rate.
• Establish that the regularizer is strongly convex when canonical correlation λ₁ < 1, ensuring effective complexity control.

Experimental results

Research questions

• RQ1 Can LUPI and knowledge distillation be formally unified under a single multi-view learning framework?
• RQ2 Under what conditions does encouraging agreement between student and teacher predictors lead to faster convergence?
• RQ3 How does the use of canonical correlation analysis (CCA) contribute to complexity control in the hypothesis space?
• RQ4 What is the impact of optimistic assumptions about teacher and student performance on convergence rates?
• RQ5 Can a simple, differentiable regularizer based on prediction discrepancy outperform standard ERM in both LUPI and distillation?

Key findings

• The proposed RERM framework with a prediction discrepancy regularizer achieves a faster O(1/n) convergence rate than standard ERM under optimistic assumptions.
• The effective hypothesis space is reduced due to agreement between views, which leads to improved generalization and faster learning.
• When canonical correlation λ₁ < 1, the regularizer ensures strong convexity in both student and teacher parameters, enabling tighter generalization bounds.
• The complexity of the hypothesis class is effectively controlled by the variance of the privileged view (S²), especially when S² ≪ B², as shown in Lemma 10.
• The derived bound on expected loss is O(α/n + √(αL*/n)), where α is a function of model and data properties, demonstrating improved convergence over baseline methods.
• The method generalizes beyond paired data and can be extended to settings with unpaired or imbalanced labeled data.