[Paper Review] Return of the features. Efficient feature selection and interpretation for photometric redshifts

Antonio D’Isanto, S. Cavuoti+2 more|arXiv (Cornell University)|Mar 27, 2018

Remote Sensing in Agriculture49 references3 citations

TL;DR

This paper proposes a forward feature selection method to identify high-performing, physically interpretable features for photometric redshift estimation in quasars. By generating 4,520 synthetic features from SDSS data and using k-NN and random forest models, the method discovers unexpected yet superior features that significantly improve regression accuracy over classic photometric features, with enhanced performance across redshift ranges due to complementary feature distributions.

ABSTRACT

The explosion of data in recent years has generated an increasing need for new analysis techniques in order to extract knowledge from massive datasets. Machine learning has proved particularly useful to perform this task. Fully automatized methods have recently gathered great popularity, even though those methods often lack physical interpretability. In contrast, feature based approaches can provide both well-performing models and understandable causalities with respect to the correlations found between features and physical processes. Efficient feature selection is an essential tool to boost the performance of machine learning models. In this work, we propose a forward selection method in order to compute, evaluate, and characterize better performing features for regression and classification problems. Given the importance of photometric redshift estimation, we adopt it as our case study. We synthetically created 4,520 features by combining magnitudes, errors, radii, and ellipticities of quasars, taken from the SDSS. We apply a forward selection process, a recursive method in which a huge number of feature sets is tested through a kNN algorithm, leading to a tree of feature sets. The branches of the tree are then used to perform experiments with the random forest, in order to validate the best set with an alternative model. We demonstrate that the sets of features determined with our approach improve the performances of the regression models significantly when compared to the performance of the classic features from the literature. The found features are unexpected and surprising, being very different from the classic features. Therefore, a method to interpret some of the found features in a physical context is presented. The methodology described here is very general and can be used to improve the performance of machine learning models for any regression or classification task.

Motivation & Objective

• Address the challenge of improving photometric redshift estimation accuracy in the era of big astronomical data.
• Overcome the limitations of black-box deep learning models by developing a physically interpretable, feature-based machine learning approach.
• Identify and validate a set of high-performing, non-intuitive features that outperform traditional photometric features.
• Demonstrate the stability and generalizability of the feature selection methodology across different model architectures.
• Enable better physical interpretation of feature importance in relation to quasar redshift and intrinsic properties.

Proposed method

• Synthetically generate 4,520 features by combining magnitudes, errors, radii, and ellipticities from SDSS quasar data.
• Apply a forward selection algorithm to iteratively build and evaluate feature subsets using k-Nearest-Neighbours (k-NN) regression.
• Construct a feature tree from the k-NN evaluation process, where each branch represents a candidate feature set.
• Validate the best-performing feature sets using random forest regression to ensure robustness and generalization.
• Use importance analysis and redshift-dependent feature evaluation to interpret the physical relevance of selected features.
• Compare the performance and distribution of selected features against the classic 10 photometric features across redshift bins.

Experimental results

Research questions

• RQ1 Can a systematic, forward feature selection method identify feature sets that significantly outperform classic photometric features in redshift regression?
• RQ2 What physical insights can be derived from the structure and distribution of the newly discovered features?
• RQ3 How stable and generalizable are the selected features across different model architectures and redshift ranges?
• RQ4 To what extent do the selected features capture complementary physical information across the redshift distribution?
• RQ5 Can the proposed method balance high performance with physical interpretability in a way that deep learning models often fail to achieve?

Key findings

• The proposed feature selection method significantly improves photometric redshift regression performance compared to the classic 10 features, with superior accuracy across all redshift ranges.
• The selected features are highly non-intuitive and structurally different from classic features, indicating discovery of novel, high-signal combinations.
• The best-performing feature sets are stable across multiple runs, with consistent distribution patterns and minimal variation in model performance.
• The selected features fill the redshift space in a complementary manner, each contributing unique information in distinct redshift regimes.
• Feature importance analysis reveals that the new features capture underlying physical processes more effectively than classic features, which fail to concentrate information as efficiently.
• The method’s performance advantage stems not from individual features, but from their collective distribution and synergy within the selected set.