[Paper Review] Selection via Proxy: Efficient Data Selection for Deep Learning

Cody Coleman, Christopher Yeh+6 more|arXiv (Cornell University)|Jun 26, 2019

Machine Learning and AlgorithmsComputer Science54 references76 citations

TL;DR

Selection via Proxy (SVP) uses small, fast proxy models to perform data selection for active learning and core-set selection in deep learning, achieving large speed-ups with minimal loss in final accuracy across multiple datasets.

ABSTRACT

Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10x faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1.6x end-to-end training time improvement.

Motivation & Objective

• Motivate data selection methods (active learning and core-set selection) for deep learning and address their high computational costs.
• Propose SVP to replace expensive target-model representations with cheaper proxy representations for selection.
• Demonstrate that proxy-based selection preserves final accuracy while substantially reducing data selection time across multiple datasets.
• Show empirical evidence of ranking correlation between proxy and target models to justify using proxies in selection processes.

Proposed method

• Create cheap proxy models by scaling down depth/width or training for fewer epochs to approximate the target’s decision boundary.
• Replace target-model representations with proxy representations when computing selection metrics (uncertainty, distance-based diversities, forgetting events).
• Apply SVP to two data selection paradigms: (i) active learning with least confidence and greedy k-centers, (ii) core-set selection with forgetting events, entropy, and greedy k-centers.
• Compare selections against targets trained on full data to assess impact on final test error.
• Evaluate correlations (Spearman/Pearson) between proxy and target rankings to explain effectiveness of proxies.
• Use datasets CIFAR-10/100, ImageNet, Amazon Review Polarity, and Amazon Review Full with models such as ResNet variants and text classifiers as proxies and targets.

Experimental results

Research questions

• RQ1 Can small proxy models provide reliable rankings for selecting informative data points compared to large target models?
• RQ2 What data selection speed-ups (in runtime) can SVP achieve across active learning and core-set tasks?
• RQ3 Do proxy-based selections maintain similar final test accuracy to target-model-based selections across diverse datasets and modalities?
• RQ4 How well do ranking signals (uncertainty, forgetting events, entropy, k-centers) correlate between proxy and target models?
• RQ5 Is SVP broadly applicable across architectures and tasks beyond image classification?

Key findings

• SVP achieves up to 41.9x speed-up in data selection runtime for Amazon Review Polarity and Full and up to 7x for CIFAR-10/100 in active learning.
• SVP yields negligible final accuracy loss, often within 0.1% of the baseline target-model selection across tasks.
• Core-set selection with proxies can remove up to 50% of CIFAR-10 data with no significant drop in ResNet164 accuracy, achieving about 1.6x end-to-end training speed-up.
• Proxy models trained for fewer epochs or with smaller architectures correlate highly with large target models in ranking examples for uncertainty, forgetting events, and k-centers.
• Across datasets and architectures, proxy rankings show high Spearman’s/Pearson correlations with large models, supporting broad applicability of SVP.