[Paper Review] Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value

Yongchan Kwon, James Zou|arXiv (Cornell University)|Apr 16, 2023

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes Data-OOB, a computationally efficient data valuation method for bagging models that leverages out-of-bag (OOB) estimates to assign data values without retraining models. It achieves sub-2.25-hour inference on 1 million samples using a single CPU and outperforms state-of-the-art methods in identifying mislabeled and influential data points.

ABSTRACT

Data valuation is a powerful framework for providing statistical insights into which data are beneficial or detrimental to model training. Many Shapley-based data valuation methods have shown promising results in various downstream tasks, however, they are well known to be computationally challenging as it requires training a large number of models. As a result, it has been recognized as infeasible to apply to large datasets. To address this issue, we propose Data-OOB, a new data valuation method for a bagging model that utilizes the out-of-bag estimate. The proposed method is computationally efficient and can scale to millions of data by reusing trained weak learners. Specifically, Data-OOB takes less than 2.25 hours on a single CPU processor when there are $10^6$ samples to evaluate and the input dimension is 100. Furthermore, Data-OOB has solid theoretical interpretations in that it identifies the same important data point as the infinitesimal jackknife influence function when two different points are compared. We conduct comprehensive experiments using 12 classification datasets, each with thousands of sample sizes. We demonstrate that the proposed method significantly outperforms existing state-of-the-art data valuation methods in identifying mislabeled data and finding a set of helpful (or harmful) data points, highlighting the potential for applying data values in real-world applications.

Motivation & Objective

• Address the computational infeasibility of Shapley-based data valuation methods on large-scale datasets.
• Develop a data valuation framework that is both efficient and statistically interpretable for real-world machine learning applications.
• Leverage existing weak learners in bagging models (e.g., random forests) to avoid retraining and reduce computational cost.
• Ensure consistency with established influence measures like the infinitesimal jackknife by proving order consistency under mild assumptions.
• Demonstrate superior performance in identifying helpful, harmful, and mislabeled data points across diverse classification datasets.

Proposed method

• Utilizes out-of-bag (OOB) estimates from bootstrap sampling in bagging models to compute data values without retraining.
• Assigns a data value as the average of model performance scores evaluated on OOB samples across all bootstrap iterations.
• Reuses already trained weak learners (e.g., decision trees in random forests), enabling O(1) inference cost per data point after model training.
• Employs a closed-form approximation of influence by aggregating OOB predictions, avoiding the need for full model retraining.
• Theoretical foundation: proves that Data-OOB preserves the same ranking of data points as the infinitesimal jackknife influence function under mild regularity conditions.
• Applies to any tabular dataset where random forests are applicable, making it broadly deployable in data-centric ML workflows.

Figure 1: Illustration of the proposed data valuation method. The OOB stands for the out-of-bag set. For each bootstrap sampling procedure, we evaluate an estimate $T_{b}(\star)$ if the datum $\star$ is in the OOB set. Here, $T_{b}(\star)$ is a score of the model trained with the $b$ -th bootstrap d

Experimental results

Research questions

• RQ1 Can a data valuation method based on OOB estimates in bagging models achieve both computational efficiency and statistical interpretability?
• RQ2 Does Data-OOB maintain consistency with established influence measures like the infinitesimal jackknife in ranking data points?
• RQ3 How does Data-OOB compare to state-of-the-art data valuation methods in identifying mislabeled and influential data points?
• RQ4 Can Data-OOB scale to large datasets (e.g., 1M samples) with minimal computational overhead?
• RQ5 How robust is Data-OOB to variations in the number of weak learners (B) in the ensemble?

Key findings

• Data-OOB computes data values in under 2.25 hours on a single CPU for a dataset of 1 million samples with 100 features, significantly outperforming model-retraining-based methods.
• The method achieves stable F1-scores for mislabeled data detection across different numbers of weak learners (B ∈ {400, 800, 3200}), indicating robustness to hyperparameter choice.
• In experiments across 12 classification datasets, Data-OOB outperforms KNN-Shapley and AME in identifying helpful and harmful data points, as shown by distribution shifts after data removal.
• Data-OOB demonstrates order consistency with the infinitesimal jackknife influence function, meaning it ranks the same data point as more influential when comparing two points.
• The method identifies mislabeled data more effectively than existing state-of-the-art approaches, highlighting its utility in data curation and data quality assessment.
• Data-OOB is insensitive to the number of weak learners B, showing stable performance across different ensemble sizes, which enhances practical usability.

Figure 2: Elapsed time comparison between KNN Shapley , AME , and Data-OOB . We use a synthetic binary classification dataset with (left) $d=10$ and (right) $d=100$ . We exclude the setting $(n,d)=(5\times 10^{5},100)$ as KNN Shapley exceeds 24 hours. The error bar indicates a 95% confidence interva