[Paper Review] Data Cleansing for Models Trained with SGD

Satoshi Hara, Atsushi Nitanda+1 more|arXiv (Cornell University)|Jun 1, 2019

Machine Learning and Data Classification14 citations

TL;DR

This paper proposes a domain-agnostic data cleansing method for models trained with stochastic gradient descent (SGD), which identifies influential training instances by retracing SGD steps and leveraging intermediate models. The approach enables non-experts to improve model accuracy without prior knowledge, achieving significant performance gains on MNIST and CIFAR10 by removing identified influential instances.

ABSTRACT

Data cleansing is a typical approach used to improve the accuracy of machine learning models, which, however, requires extensive domain knowledge to identify the influential instances that affect the models. In this paper, we propose an algorithm that can identify influential instances without using any domain knowledge. The proposed algorithm automatically cleans the data, which does not require any of the users' knowledge. Hence, even non-experts can improve the models. The existing methods require the loss function to be convex and an optimal model to be obtained, which is not always the case in modern machine learning. To overcome these limitations, we propose a novel approach specifically designed for the models trained with stochastic gradient descent (SGD). The proposed method infers the influential instances by retracing the steps of the SGD while incorporating intermediate models computed in each step. Through experiments, we demonstrate that the proposed method can accurately infer the influential instances. Moreover, we used MNIST and CIFAR10 to show that the models can be effectively improved by removing the influential instances suggested by the proposed method.

Motivation & Objective

• To address the limitation of existing data cleansing methods that require domain expertise to identify influential training instances.
• To overcome the reliance of prior methods on convex loss functions and optimal model convergence, which are often unmet in modern deep learning.
• To develop a method specifically tailored for models trained with stochastic gradient descent (SGD), a standard training paradigm in modern machine learning.
• To enable non-experts to improve model performance through automated data cleansing without manual intervention or domain knowledge.
• To demonstrate the effectiveness of removing identified influential instances in improving model accuracy on standard benchmarks.

Proposed method

• The method infers influential training instances by retracing the optimization path of stochastic gradient descent (SGD) across all training steps.
• It leverages intermediate models computed at each SGD update step to assess the impact of individual training instances on model behavior.
• The approach does not require the loss function to be convex or the final model to be optimal, making it applicable to real-world deep learning settings.
• Influential instances are identified based on their contribution to model updates along the training trajectory, using a trace-based influence estimation technique.
• The method operates entirely in a data-driven, automatic fashion, eliminating the need for user-defined heuristics or domain-specific criteria.
• After identifying influential instances, the method evaluates model improvement by removing them and retraining on the cleaned dataset.

Experimental results

Research questions

• RQ1 Can a data cleansing method identify influential training instances without requiring domain knowledge or convexity assumptions in SGD-trained models?
• RQ2 How accurately can the proposed method infer the most influential training instances compared to existing influence estimation techniques?
• RQ3 To what extent does removing the identified influential instances improve model generalization and accuracy on standard benchmarks?
• RQ4 Can the method be effectively applied to real-world deep learning models trained with SGD, even when the loss is non-convex and the final model is suboptimal?
• RQ5 Does the proposed method enable non-experts to improve model performance through automated data cleansing?

Key findings

• The proposed method successfully identifies influential training instances in models trained with SGD without requiring domain knowledge or convex loss functions.
• The method achieves high accuracy in influence estimation by leveraging intermediate models from each SGD update step.
• On the MNIST dataset, removing the top influential instances identified by the method leads to measurable improvements in model test accuracy.
• On the CIFAR-10 dataset, the proposed cleansing approach results in a significant reduction in test error after retraining on the cleaned data.
• The method enables non-experts to improve model performance through automated data cleansing, demonstrating practical usability in real-world machine learning workflows.
• The approach is robust to non-convex optimization landscapes and does not require convergence to an optimal model, making it suitable for modern deep learning applications.