[Paper Review] Overfitting and correlations in model fitting with separation ratios

I. W. Roxburgh|arXiv (Cornell University)|Aug 22, 2018

Stellar, planetary, and galactic studies3 citations

TL;DR

This paper demonstrates that combining separation ratios $r_{01}$ and $r_{10}$ into a single $r_{010}$ sequence causes overfitting due to strong correlations, leading to nearly singular covariance matrices with high condition numbers and spurious model comparisons. The authors advocate combining $r_{10}$, $r_{01}$, and $r_{02}$ into a non-overfitted $r_{102}$ or $r_{012}$ sequence using properly derived covariance matrices, which yields consistent and reliable model fits.

ABSTRACT

The $r_{01}$ and $r_{10}$ separation ratios are not independent so combing them into a single series $r_{010}$ is overfitting the data, this can lead to almost singular covariance matrices with very large condition numbers, and hence to spurious results when comparing models and observations. Since the $r_{02}$ ratios are strongly correlated with $r_{10}$ and $r_{01}$ ratios, they should be combined into a single series $r_{102}$ (or $r_{012}$), which are not overfitted, and models and observation compared using the covariance matrix $cov_{102}$ (or $cov_{012}$) of the combined set. I illustrate these points by comparing the revised Legacy Project data with my results on the 10 Kepler stars in common.

Motivation & Objective

• To identify the risk of overfitting when combining $r_{01}$ and $r_{10}$ ratios into a single $r_{010}$ sequence in asteroseismic model fitting.
• To demonstrate that the resulting covariance matrices for $r_{010}$ become nearly singular with large condition numbers, leading to spurious model comparison results.
• To show that $r_{02}$ ratios are strongly correlated with $r_{01}$ and $r_{10}$, necessitating their joint treatment to avoid overfitting.
• To propose combining $r_{10}$, $r_{01}$, and $r_{02}$ into a single non-overfitted sequence $r_{102}$ or $r_{012}$ for more robust model-data comparisons.
• To validate the proposed method using 10 Kepler stars common to the Legacy Project and the author’s analysis, showing consistent $χ^2$ values when using $r_{102}$ or $r_{012}$

Proposed method

• Derive the $r_{01}$, $r_{10}$, and $r_{02}$ separation ratios from observed frequencies using standard definitions involving frequency differences and normalization.
• Compute the covariance matrix of each ratio using the delta method: $cov(r_i, r_j) = \sum_{k,m} \frac{\partial r_i}{\partial \nu_k} cov(\nu_k, \nu_m) \frac{\partial r_j}{\partial \nu_m}$, based on frequency covariance matrices.
• Generate combined sequences $r_{010}$, $r_{102}$, and $r_{012}$ from individual ratios, treating them as multivariate data with full covariance structure.
• Assess model fit quality using $\chi^2$ statistics computed with the full covariance matrix of the combined sequence.
• Compare results using both the Legacy Project’s MCMC-derived covariance matrices and the author’s independently derived matrices from maximum-likelihood estimation.
• Evaluate condition numbers of the resulting covariance matrices to detect near-singularity due to overfitting.

Experimental results

Research questions

• RQ1 Does combining $r_{01}$ and $r_{10}$ into a single $r_{010}$ sequence lead to overfitting and unstable covariance matrices in asteroseismic model fitting?
• RQ2 Why do $\chi^2$ values for $r_{010}$ sequences become anomalously large compared to $r_{01}$ and $r_{10}$ when using the same data and errors?
• RQ3 Are $r_{02}$ ratios sufficiently correlated with $r_{01}$ and $r_{10}$ to justify their combination into a single sequence like $r_{102}$ or $r_{012}$?
• RQ4 Can the use of combined sequences $r_{102}$ or $r_{012}$ with properly computed covariance matrices yield consistent and reliable model fits compared to individual ratios?
• RQ5 How do the condition numbers of the covariance matrices of $r_{010}$, $r_{102}$, and $r_{012}$ compare, and what does this imply for numerical stability in model comparison?

Key findings

• The $r_{010}$ sequence, formed by combining $r_{01}$ and $r_{10}$, exhibits $\chi^2$ values as high as 5.6 for some stars (e.g., KIC 8379927), indicating poor fit, while individual $r_{01}$ and $r_{10}$ sequences show $\chi^2 \sim 10^{-3}$, indicating excellent agreement.
• For KIC 6106415, the $\chi^2_{\text{cov}}$ for $r_{010}$ is 4.1139, while for $r_{01}$ and $r_{10}$ it is only 0.0002 and 0.0003, respectively, demonstrating a significant anomaly due to overfitting.
• The $r_{102}$ and $r_{012}$ sequences, formed by combining $r_{10}$, $r_{01}$, and $r_{02}$, yield $\chi^2_{\text{cov}}$ values that are consistent with the average of the individual $r_{10}$ and $r_{02}$ fits, indicating no overfitting.
• The condition number of the $r_{010}$ covariance matrix is extremely high (implied by near-singularity), whereas the $r_{102}$ and $r_{012}$ matrices have reasonable condition numbers, confirming numerical stability.
• For KIC 12069449, the $\chi^2_{\text{cov}}$ for $r_{0100}$ reaches 27.518, while $r_{102}$ and $r_{012}$ values are 4.883 and 3.965, respectively, showing that $r_{010}$ is highly unstable.
• The $r_{102}$ and $r_{012}$ sequences produce consistent model fits across all 10 Kepler stars, with $\chi^2_{\text{cov}}$ values for $r_{102}$ closely matching the average of $r_{10}$ and $r_{02}$, confirming that these sequences are not overfitted.