[Paper Review] Two New Estimators of Entropy for Testing Normality

Akram Kohansal, Saeid Rezakhah|arXiv (Cornell University)|May 1, 2014

Statistical Distribution Estimation and Applications23 references3 citations

TL;DR

This paper introduces two new entropy estimators—HVE^R_mn and HW^R_mn—by refining Van Es’s and Wieczorkowski & Grzegorewski’s estimators using ranked set sampling, achieving lower bias and root mean squared error (RMSE). The proposed estimators enable two highly powerful normality tests, with HW^R_mn showing superior performance, especially at small sample sizes (n=15), outperforming existing tests even at n=45.

ABSTRACT

We present two new estimators for estimating the entropy of absolutely continuous random variables. Some properties of them are considered, specifically consistency of the first is proved. The introduced estimators are compared with the existing entropy estimators. Also, we propose two new tests for normality based on the introduced entropy estimators and compare their powers with the powers of other tests for normality. The results show that the proposed estimators and test statistics perform very well in estimating entropy and testing normality. A real example is presented and analyzed.

Motivation & Objective

• To develop entropy estimators with reduced bias and RMSE for continuous distributions.
• To improve existing entropy-based normality tests by leveraging ranked set sampling (RSS) and modified estimators.
• To evaluate the performance of new test statistics against established normality tests using Monte Carlo simulations.
• To provide a practical, empirically validated method for estimating test power in entropy-based goodness-of-fit testing.

Proposed method

• Modified Van Es’s entropy estimator (HVE_mn) by incorporating ranked set sampling (RSS) to improve estimation accuracy.
• Modified Wieczorkowski & Grzegorewski’s estimator (HW_mn) using RSS and bias correction via the digamma function.
• Proposed two new test statistics, TVE^R_mn and TW^R_mn, based on the refined estimators for normality testing.
• Used Monte Carlo simulations to compare the power of the new tests with established ones (e.g., Kolmogorov-Smirnov, Anderson-Darling, Shapiro-Wilk).
• Applied an empirical power estimation procedure involving simulation of inverse Gaussian samples under estimated parameters.
• Validated results using a real dataset of active repair times, fitting an inverse Gaussian distribution and assessing p-values and power.

Experimental results

Research questions

• RQ1 Can ranked set sampling improve the bias and RMSE of existing entropy estimators for continuous distributions?
• RQ2 How do the new entropy estimators HVE^R_mn and HW^R_mn compare in performance to Vasicek’s, Van Es’s, Correa’s, and others in terms of bias and RMSE?
• RQ3 Do the new test statistics TVE^R_mn and TW^R_mn exhibit higher statistical power than established normality tests, especially at small sample sizes?
• RQ4 Is the proposed empirical power estimation procedure effective and reliable for comparing entropy-based test statistics?
• RQ5 Can the new estimators and tests be effectively applied to real-world data, such as active repair times, with meaningful p-values and power?

Key findings

• The proposed estimator HW^R_mn consistently exhibits the lowest bias and RMSE among all compared entropy estimators across various distributions.
• The test statistic TW^R_mn achieved a power of 0.866 at n=15, significantly outperforming competitors with n=45, which had powers of 0.5368 (TV_mn), 0.1965 (TVE_mn), and 0.4015 (TC_mn).
• The p-value for TW^R_mn on the active repair times dataset was 0.0028, indicating strong evidence against normality, with a good fit to the inverse Gaussian distribution.
• The empirical power estimation procedure successfully quantified and compared test performance, confirming the superiority of the new test statistics.
• The new estimators and tests are robust and effective, with HW^R_mn showing the best overall performance in simulation studies.
• The results confirm that ranked set sampling significantly enhances the efficiency of entropy estimation and normality testing, particularly in small-sample settings.