[Paper Review] Sensitivity Analysis of Simulation-Based Inference for Galaxy Clustering

Chirag Modi, Shivam Pandey+4 more|arXiv (Cornell University)|Sep 26, 2023

demographic modeling and climate adaptation4 citations

TL;DR

This paper conducts a sensitivity analysis of simulation-based inference (SBI) for galaxy clustering, evaluating how variations in gravity models, halo-finders (FoF vs. Rockstar), and galaxy-halo occupation distribution (HOD) models affect inference of cosmological parameters σ₈ and Ωₘ. It finds that SBI is robust to gravity model changes but highly sensitive to halo-finder choice—Rockstar introduces up to ~20% bias in bispectrum-based inference—while HOD complexity significantly impacts robustness, with simpler models failing when applied to complex HOD data.

ABSTRACT

Simulation-based inference (SBI) is a promising approach to leverage high fidelity cosmological simulations and extract information from the non-Gaussian, non-linear scales that cannot be modeled analytically. However, scaling SBI to the next generation of cosmological surveys faces the computational challenge of requiring a large number of accurate simulations over a wide range of cosmologies, while simultaneously encompassing large cosmological volumes at high resolution. This challenge can potentially be mitigated by balancing the accuracy and computational cost for different components of the the forward model while ensuring robust inference. To guide our steps in this, we perform a sensitivity analysis of SBI for galaxy clustering on various components of the cosmological simulations: gravity model, halo-finder and the galaxy-halo distribution models (halo-occupation distribution, HOD). We infer the $σ_8$ and $Ω_m$ using galaxy power spectrum multipoles and the bispectrum monopole assuming a galaxy number density expected from the luminous red galaxies observed using the Dark Energy Spectroscopy Instrument (DESI). We find that SBI is insensitive to changing gravity model between $N$-body simulations and particle mesh (PM) simulations. However, changing the halo-finder from friends-of-friends (FoF) to Rockstar can lead to biased estimate of $σ_8$ based on the bispectrum. For galaxy models, training SBI on more complex HOD leads to consistent inference for less complex HOD models, but SBI trained on simpler HOD models fails when applied to analyze data from a more complex HOD model. Based on our results, we discuss the outlook on cosmological simulations with a focus on applying SBI approaches to future galaxy surveys.

Motivation & Objective

• To assess the robustness of simulation-based inference (SBI) for galaxy clustering under variations in key components of cosmological simulations.
• To identify which simulation model components—gravity, halo-finder, or galaxy-halo occupation distribution (HOD)—most significantly impact inference accuracy for σ₈ and Ωₘ.
• To evaluate whether SBI trained on simplified models can reliably infer parameters when applied to data from more complex models.
• To guide future SBI applications to next-generation galaxy surveys by identifying critical sources of model misspecification.
• To emphasize the need for end-to-end sensitivity analysis over isolated component testing, especially for higher-order statistics like the bispectrum.

Proposed method

• Perform SBI using high-fidelity cosmological simulations with varying gravity models (N-body vs. particle mesh), halo-finders (FoF vs. Rockstar), and HOD parameterizations.
• Train neural density estimators on simulated galaxy power spectrum multipoles and bispectrum monopole to infer posterior distributions over σ₈ and Ωₘ.
• Use a fixed galaxy number density matching DESI’s luminous red galaxies to ensure realistic observational scaling.
• Apply end-to-end inference pipelines to evaluate sensitivity across simulation components, focusing on posterior bias and consistency.
• Compare results across different summary statistics (power spectrum and bispectrum) to assess sensitivity to non-Gaussian features.
• Use the Quijote and FastPM simulation datasets to enable controlled, reproducible sensitivity testing.

Figure 1: Comparison of summary statistics for different forward models : We show the ratio of summary statistics for galaxy catalogs generated by varying one stage of the forward model, as indicated by the title of columns, while keeping the other two stages fixed. The three rows show the ratios fo

Experimental results

Research questions

• RQ1 How does changing the gravity model (N-body vs. PM) affect SBI inference of σ₈ and Ωₘ?
• RQ2 What is the impact of switching from FoF to Rockstar halo-finder on the accuracy of SBI for galaxy clustering statistics?
• RQ3 How does the complexity of the HOD model influence the robustness of SBI when trained on simple HOD but applied to complex HOD data?
• RQ4 Are higher-order statistics like the bispectrum more sensitive to simulation model misspecification than the power spectrum?
• RQ5 Can end-to-end SBI pipelines reveal non-intuitive sensitivities, such as insensitivity to gravity model but sensitivity to halo-finder choice?

Key findings

• SBI inference is insensitive to the choice between N-body and particle mesh (PM) gravity models, with no significant bias in σ₈ or Ωₘ estimates.
• Switching from FoF to Rockstar halo-finder introduces up to ~20% bias in σ₈ estimates when using the bispectrum, though power spectrum inference remains robust.
• SBI trained on complex HOD models produces consistent posteriors when applied to data from simpler HOD models, but the reverse is not true—SBI trained on simple HOD fails on complex HOD data.
• The bispectrum is more sensitive to model misspecification than the power spectrum, with biased results arising from halo-finder changes even when power spectrum inference remains unbiased.
• End-to-end sensitivity analysis reveals non-trivial dependencies, such as insensitivity to gravity model but strong sensitivity to halo-finder, highlighting the importance of holistic pipeline testing.
• Robustness of SBI is increasingly challenged by higher-order statistics and model misspecification, necessitating validation on complex, multi-model simulation data for future surveys.

(a) Residuals and 1- $\sigma$ posterior width in inferring $\Omega_{m}$ and $\sigma_{8}$ (columns) for 100 different simulations with the power spectrum (top row) and bispectrum (bottom row) statistic using SBI. The labels indicate the forward model used for training. In blue we show results for SBI