[Paper Review] Direct optimisation of the discovery significance when training neural networks to search for new physics in particle colliders

A. Elwood, D. Krücker|arXiv (Cornell University)|Jan 1, 2018

Particle physics theoretical and experimental studies12 references10 citations

TL;DR

This paper introduces two novel neural network loss functions—optimized for the s/√(s+b) and Asimov significance estimates—that directly maximize statistical discovery significance in particle physics searches. When trained on a simulated LHC SUSY search with 30 fb⁻¹ of 14 TeV data, the Asimov-based loss outperformed binary cross entropy, especially under high systematic uncertainties, by achieving a 9.9 ± 1.6σ significance compared to 6.4 ± 0.7σ for standard cross entropy, due to higher signal purity in the selected region.

ABSTRACT

We introduce two new loss functions designed to directly optimise the statistical significance of the expected number of signal events when training neural networks to classify events as signal or background in the scenario of a search for new physics at a particle collider. The loss functions are designed to directly maximise commonly used estimates of the statistical significance, $s/\sqrt{s+b}$, and the Asimov estimate, $Z_A$. We consider their use in a toy SUSY search with 30~fb$^{-1}$ of 14~TeV data collected at the LHC. In the case that the search for the SUSY model is dominated by systematic uncertainties, it is found that the loss function based on $Z_A$ can outperform the binary cross entropy in defining an optimal search region.

Motivation & Objective

• To address the mismatch between standard classification loss functions (e.g., binary cross entropy) and the true goal of new physics searches: maximizing statistical significance, not accuracy.
• To develop loss functions that directly optimize the expected discovery significance, specifically s/√(s + b) and the Asimov Z estimate, during neural network training.
• To evaluate whether these significance-optimized losses improve performance in scenarios dominated by systematic uncertainties, particularly in challenging compressed SUSY models.
• To demonstrate that direct optimization of significance leads to higher signal purity and better separation power than standard training objectives.

Proposed method

• Proposes two new loss functions: ℓs/√(s+b) and ℓAsimov, which directly optimize the expected statistical significance of signal over background.
• The losses are computed using estimated signal (s) and background (b) event counts derived from the network’s output scores, with significance calculated using the Asimov formula ZA = (s + b) / √(s + b) and s/√(s + b) for comparison.
• Networks are trained using stochastic gradient descent with these significance-based losses, using a preselected dataset of 1.4 million events (50% signal, 50% background) from a simulated stop pair production search.
• The input features include low-level variables (energy, transverse momentum, pseudorapidity, etc.) of leading jets and leptons, as well as high-level variables like HT, ET/, and mT.
• A separate test sample of 600,000 events is used exclusively for evaluation to avoid data leakage.
• Performance is evaluated by scanning classifier score cuts and computing the Asimov significance with systematic uncertainties propagated via Poisson errors.

Experimental results

Research questions

• RQ1 Can direct optimization of statistical significance during neural network training improve discovery sensitivity in new physics searches?
• RQ2 Does a loss function based on the Asimov significance estimate yield better signal purity and higher significance than standard binary cross entropy, especially under systematic uncertainties?
• RQ3 How do the new loss functions perform in a compressed SUSY model where signal and background are kinematically similar but background is dominant?
• RQ4 Is the performance gain from significance-optimized losses dependent on the level of systematic uncertainty in the background?

Key findings

• In the uncompressed SUSY model (900 GeV stop, 100 GeV LSP), all three loss functions—binary cross entropy, ℓs/√(s+b), and ℓAsimov—performed similarly, as statistical uncertainty dominated the significance.
• In the compressed SUSY model (600 GeV stop, 400 GeV LSP), where systematic uncertainties dominate, the ℓAsimov loss achieved a 9.9 ± 1.6σ significance with 50% systematic uncertainty, significantly outperforming binary cross entropy’s 6.4 ± 0.7σ.
• The ℓAsimov-optimized network achieved a signal purity (s/b) of ∼30, compared to ∼6 for binary cross entropy and ∼1 for ℓs/√(s+b), indicating superior signal enrichment.
• The ℓs/√(s+b) loss performed poorly in the compressed model because it does not account for systematic uncertainties, leading to a low signal purity of only ∼1.
• The ℓAsimov loss showed robustness to cut selection, with optimal performance maintained for classifier score cuts above ∼0.5, reducing the need for hyperparameter tuning.
• Pretraining with ℓs/√(s+b) before ℓAsimov optimization significantly reduced training time, suggesting a practical training strategy.