[Paper Review] The use of neural networks to probe the structure of the nearby universe

M. Paolillo, Longo, G.+5 more|arXiv (Cornell University)|Jan 5, 2007

Spectroscopy and Chemometric Analyses3 references3 citations

TL;DR

This paper proposes a machine learning framework using multi-layer perceptron neural networks and unsupervised clustering to derive photometric redshifts and classify galaxy types in the nearby universe (z < 0.5) from Sloan Digital Sky Survey (SDSS) photometric data. It achieves a photometric redshift accuracy of σ₃ ≈ 0.0208 for main galaxies and σ₃ ≈ 0.0163 for luminous red galaxies, enabling a 3D map of the nearby cosmos and identifying morphological clusters including AGN and QSO candidates with minimal spectroscopic priors.

ABSTRACT

In the framework of the European VO-Tech project, we are implementing new machine learning methods specifically tailored to match the needs of astronomical data mining. In this paper, we shortly present the methods and discuss an application to the Sloan Digital Sky Survey public data set. In particular, we discuss some preliminary results on the 3-D taxonomy of the nearby (z &lt; 0.5) universe. Using neural networks trained on the available spectroscopic base of knowledge we derived distance estimates for ca. 30 million galaxies distributed over 8,000 sq. deg. We also use unsupervised clustering tools to investigate whether it is possible to characterize in broad morphological bins the nature of each object and produce a reliable list of candidate AGNs and QSOs.

Motivation & Objective

• To develop machine learning tools tailored for astronomical data mining in the context of the Virtual Observatory.
• To address the challenge of deriving reliable photometric redshifts for massive photometric surveys with high degeneracy and missing data.
• To classify galaxies into broad morphological types—such as normal galaxies, AGNs, and QSOs—using only photometric data and unsupervised clustering.
• To produce a 3D map of the nearby universe (z < 0.5) using photometric redshift estimates for structure detection and selection function definition.
• To validate clustering results using spectroscopic classifications from SDSS as ground truth, minimizing reliance on labeled data.

Proposed method

• A two-step neural network approach using Multi-Layer Perceptrons (MLPs) to first classify galaxies as nearby (z < 0.25) or distant (z > 0.25), then train separate MLPs for each redshift regime.
• Photometric redshifts were estimated using MLPs trained on spectroscopic redshifts in the ranges [0.0, 0.27] for nearby and [0.23, 0.50] for distant galaxies, with reliability assessed in [0.01, 0.25] and [0.25, 0.48] respectively.
• The PPS (Projection Pursuit Search) algorithm identified 614 initial clusters in the 4-color photometric space (u-g, g-r, r-i, i-z) using latent variables to capture non-Gaussian structures.
• The NEC (Neural Estimation of Clusters) algorithm agglomerated PPS clusters using Negentropy maximization to find optimal cluster partitions, with threshold tuning via plateau analysis in the 'threshold vs. number of clusters' plot.
• Clustering was performed solely on photometric data; spectroscopic class (specClass) was used only for validation and interpretation of cluster content.
• Systematic errors in photometric redshifts were corrected via interpolative correction in the test set, improving accuracy and reducing scatter.

Experimental results

Research questions

• RQ1 Can neural networks trained on spectroscopic data produce reliable photometric redshift estimates for 30 million galaxies in the nearby universe (z < 0.5)?
• RQ2 Can unsupervised clustering of photometric data alone identify distinct morphological or physical classes of galaxies, such as AGNs and QSOs, without prior labeling?
• RQ3 What is the optimal separation between nearby and distant galaxy populations in terms of photometric redshift estimation and network generalization?
• RQ4 How can non-Gaussian structure detection via Negentropy maximization improve clustering in high-dimensional, degenerate astronomical data?
• RQ5 To what extent can photometric clustering reflect underlying physical properties like galaxy type or activity, as validated by spectroscopic classification?

Key findings

• The photometric redshift estimation achieved a robust scatter of σ₃ = 0.0208 for the main galaxy sample and σ₃ = 0.0163 for the luminous red galaxy sample, with higher accuracy in the central redshift ranges.
• The two-step MLP approach improved generalization by separating nearby (z < 0.25) and distant (z > 0.25) galaxies, reducing bias in extrapolation.
• The PPS algorithm identified 614 initial clusters in the 4-color photometric space, which were refined by the NEC algorithm to a final set of 31 robust clusters.
• Cluster 1, 2, and 6 were dominated by normal galaxies (SP2), while clusters 5 and 9 showed strong AGN (SP3) dominance, indicating successful photometric identification of active galactic nuclei.
• Late-type stars (SP6) were primarily found in clusters 7 and 8, with contamination observed in cluster 10, which was otherwise dominated by normal galaxies.
• The method successfully produced a 3D map of the nearby universe using photometric redshifts, enabling future structure detection and selection function modeling.