[Paper Review] Fast Prediction of New Feature Utility

Hoyt Koepke, Mikhail Bilenko|arXiv (Cornell University)|Jun 18, 2012

Machine Learning and Data Classification24 references3 citations

TL;DR

This paper proposes a fast, retraining-free method to predict whether a new feature will improve model performance by leveraging the correlation between the feature and the loss gradient of a trained predictor. The technique enables consistent, efficient hypothesis testing for feature utility, validated on benchmarks and industrial data with significant computational savings.

ABSTRACT

We study the new feature utility prediction problem: statistically testing whether adding a new feature to the data representation can improve predictive accuracy on a supervised learning task. In many applications, identifying new informative features is the primary pathway for improving performance. However, evaluating every potential feature by re-training the predictor with it can be costly. The paper describes an efficient, learner-independent technique for estimating new feature utility without re-training based on the current predictor's outputs. The method is obtained by deriving a connection between loss reduction potential and the new feature's correlation with the loss gradient of the current predictor. This leads to a simple yet powerful hypothesis testing procedure, for which we prove consistency. Our theoretical analysis is accompanied by empirical evaluation on standard benchmarks and a large-scale industrial dataset.

Motivation & Objective

• To address the high computational cost of evaluating new features via full retraining in supervised learning.
• To develop a learner-agnostic method that predicts feature utility without retraining the predictor.
• To enable rapid screening of large pools of candidate features in real-world applications.
• To provide a statistically consistent hypothesis test for feature utility prediction.

Proposed method

• The method derives a connection between the potential reduction in training loss and the correlation between a new feature and the loss gradient of the current predictor.
• It formulates a test statistic based on this correlation, enabling a hypothesis test for feature utility without retraining.
• The approach is consistent under mild regularity conditions, ensuring reliable predictions as sample size grows.
• The technique is applicable to any differentiable loss function and any base predictor.
• It uses the current predictor's outputs to estimate the loss gradient, avoiding retraining.
• The method is computationally efficient, requiring only a single pass over the data to compute the test statistic.

Experimental results

Research questions

• RQ1 Can we predict whether a new feature will improve predictive accuracy without retraining the model?
• RQ2 How can we efficiently estimate feature utility in a way that is independent of the learning algorithm?
• RQ3 What statistical properties does the proposed method have in terms of consistency and reliability?
• RQ4 How does the method compare to full retraining in terms of accuracy and computational cost?
• RQ5 Can the method be applied effectively to both standard benchmarks and large-scale industrial datasets?

Key findings

• The proposed method achieves consistent feature utility prediction, with theoretical guarantees of correctness as sample size increases.
• Empirical evaluation shows the method closely approximates the performance of full retraining in identifying useful features.
• The method reduces computational cost by orders of magnitude compared to retraining-based evaluation.
• On standard benchmarks and a large-scale industrial dataset, the method identifies high-utility features with high precision and low false positive rates.
• The technique is robust across different types of predictors and loss functions.
• The method enables scalable feature engineering by allowing rapid screening of thousands of candidate features without retraining.