[Paper Review] Data Valuation using Reinforcement Learning

Jinsung Yoon, Sercan Ö. Arık+1 more|arXiv (Cornell University)|Sep 25, 2019

Domain Adaptation and Few-Shot Learning29 references45 citations

TL;DR

DVRL jointly learns data values with the target predictor using reinforcement learning, improving data valuation for domain adaptation, corrupted sample discovery, and robust learning across diverse datasets.

ABSTRACT

Quantifying the value of data is a fundamental problem in machine learning. Data valuation has multiple important use cases: (1) building insights about the learning task, (2) domain adaptation, (3) corrupted sample discovery, and (4) robust learning. To adaptively learn data values jointly with the target task predictor model, we propose a meta learning framework which we name Data Valuation using Reinforcement Learning (DVRL). We employ a data value estimator (modeled by a deep neural network) to learn how likely each datum is used in training of the predictor model. We train the data value estimator using a reinforcement signal of the reward obtained on a small validation set that reflects performance on the target task. We demonstrate that DVRL yields superior data value estimates compared to alternative methods across different types of datasets and in a diverse set of application scenarios. The corrupted sample discovery performance of DVRL is close to optimal in many regimes (i.e. as if the noisy samples were known apriori), and for domain adaptation and robust learning DVRL significantly outperforms state-of-the-art by 14.6% and 10.8%, respectively.

Motivation & Objective

• Motivate data valuation as a means to improve learning when data quality and domain mismatch are issues.
• Propose a meta-learning framework that jointly trains a data value estimator with the target predictor.
• Enable adaptive sample weighting to prioritize informative data for the target task.
• Demonstrate DVRL's effectiveness across domain adaptation, corrupted sample discovery, and robust learning scenarios.

Proposed method

• Introduce a data value estimator hφ that outputs selection probabilities w for each training datum (x, y).
• Train the target predictor fθ by minimizing a weighted loss using sampled data according to w.
• Model the data-sample selection as a stochastic process and optimize φ with reinforcement learning via REINFORCE using a validation-set reward.
• Use a small target-distribution validation set to provide the reinforcement signal reflecting task performance.
• Apply a moving-average baseline δ to stabilize policy gradient updates.
• Demonstrate scalability by showing DVRL’s training overhead is not exponential in dataset size.

Experimental results

Research questions

• RQ1 Can a learnable data value estimator jointly optimize with the predictor to improve target-task performance?
• RQ2 How does DVRL perform in domain adaptation, corrupted sample discovery, and robust learning compared to existing data-valuation methods?
• RQ3 What validation-set size is sufficient for reliable data valuation in various tasks?
• RQ4 Is DVRL scalable to large datasets and complex models without prohibitive computational costs?

Key findings

• DVRL yields superior data value estimates compared to permutation-based and game-theoretic baselines across image, tabular, and language datasets.
• In corrupted-sample discovery, DVRL’s ability to assign low values to noisy labels approaches optimal performance in many regimes.
• For domain adaptation and robust learning, DVRL significantly outperforms state-of-the-art methods by substantial margins (e.g., up to 14.6% and 10.8% in reported scenarios).
• DVRL remains scalable to large datasets and complex models, with training overhead about twice that of standard training rather than exponential in dataset size.