[Paper Review] Decoupling "when to update" from "how to update"

Eran Malach, Shai Shalev‐Shwartz|arXiv (Cornell University)|Jun 8, 2017

Machine Learning and Algorithms22 references127 citations

TL;DR

Proposes a meta-algorithm that updates two predictors only on disagreement to robustness against noisy labels, and demonstrates state-of-the-art results on a gender-classification task using noisy labels.

ABSTRACT

Deep learning requires data. A useful approach to obtain data is to be creative and mine data from various sources, that were created for different purposes. Unfortunately, this approach often leads to noisy labels. In this paper, we propose a meta algorithm for tackling the noisy labels problem. The key idea is to decouple "when to update" from "how to update". We demonstrate the effectiveness of our algorithm by mining data for gender classification by combining the Labeled Faces in the Wild (LFW) face recognition dataset with a textual genderizing service, which leads to a noisy dataset. While our approach is very simple to implement, it leads to state-of-the-art results. We analyze some convergence properties of the proposed algorithm.

Motivation & Objective

• Motivate the problem of noisy labels in supervised deep learning and the need for robust data labeling methods.
• Propose a meta-algorithm that decouples the timing of updates from the update rule to mitigate label noise.
• Ground the approach theoretically in convergence properties for a perceptron-based setting.
• Demonstrate empirical effectiveness on real-world noisy-label data (gender classification from LFW using a name-based gender service).
• Discuss limitations and avenues for distribution-dependent convergence and extension beyond linear predictors.

Proposed method

• Introduce a meta-algorithm that maintains two predictors from the same hypothesis class.
• Perform updates only on examples where the two predictors disagree (Update by Disagreement).
• Base the updates on a standard update rule (e.g., perceptron or SGD with momentum) applied to the disagreeing subset.
• Provide pseudo-code for the Update by Disagreement procedure.
• Relate the approach to active learning and query-by-committee, noting practical differences (no full version-space maintenance).
• Analyze convergence theoretically in a linearly separable, noisy-label setting with the perceptron as the base learner.

Experimental results

Research questions

• RQ1 Does the Update by Disagreement algorithm converge under label noise?
• RQ2 How does the convergence rate compare to the noise-free perceptron, and under what assumptions?
• RQ3 What is the impact of initialization on convergence quality and optimality?
• RQ4 Can the approach yield robust performance on real-world noisy-label datasets beyond synthetic settings?
• RQ5 What are the limitations of convergence guarantees for non-linear predictors and real data distributions?

Key findings

• The algorithm converges with an update count within a constant factor of the noise-free perceptron under their theoretical model.
• Initialization matters; identical initial predictors yield no updates, and random initialization may yield suboptimal convergence without further assumptions.
• Under certain distributions, the method may not converge to zero error even with linearly separable data, unless additional initialization strategies are used.
• Empirical results on LFW-based gender classification with noisy labels show state-of-the-art robustness compared to competing methods, across different noise levels.
• When combined with a noise-adaptation and related methods (e.g., s-model), the proposed approach can achieve improvements in performance, especially as noise decreases.
• Controlled experiments indicate resilience to high label-noise rates (up to 40%), and empirical evidence suggests good convergence properties on natural data.