[Paper Review] Credibility evaluation of income data with hierarchical correlation reconstruction

Jarek Duda, Adam Szulc|arXiv (Cornell University)|Dec 19, 2018

Advanced Data Compression Techniques9 references4 citations

TL;DR

This paper proposes a hierarchical correlation reconstruction (HCR) method to evaluate the credibility of declared income by modeling its conditional probability distribution using endogenous variables. By normalizing income to a uniform [0,1] distribution and modeling the density as a linear combination of orthonormal polynomials, the approach enables interpretable, moment-based credibility scoring—with m=4 yielding optimal log-likelihood performance.

ABSTRACT

In situations like tax declarations or analyzes of household budgets we would like to automatically evaluate credibility of exogenous variable (declared income) based on some available (endogenous) variables - we want to build a model and train it on provided data sample to predict (conditional) probability distribution of exogenous variable based on values of endogenous variables. Using Polish household budget survey data there will be discussed simple and systematic adaptation of hierarchical correlation reconstruction (HCR) technique for this purpose, which allows to combine interpretability of statistics with modelling of complex densities like in machine learning. For credibility evaluation we normalize marginal distribution of predicted variable to $ρ\approx 1$ uniform distribution on $[0,1]$ using empirical distribution function $(x=EDF(y)\in[0,1])$, then model density of its conditional distribution $( extrm{Pr}(x_0|x_1 x_2\ldots))$ as a linear combination of orthonormal polynomials using coefficients modelled as linear combinations of features of the remaining variables. These coefficients can be calculated independently, have similar interpretation as cumulants, additionally allowing to directly reconstruct probability distribution. Values corresponding to high predicted density can be considered as credible, while low density suggests disagreement with statistics of data sample, for example to mark for manual verification a chosen percentage of data points evaluated as the least credible.

Motivation & Objective

• To develop a method for automatically assessing the credibility of declared income using auxiliary household variables.
• To model the full conditional probability distribution of income rather than just its mean, enabling detection of multimodal or anomalous patterns.
• To combine interpretability of statistical moments with the flexibility of machine learning-like density modeling.
• To use empirical distribution function normalization to ensure uniform marginal distributions, enabling fair credibility scoring across the income range.
• To identify low-density regions in predicted distributions as potential outliers or errors requiring manual review.

Proposed method

• Normalize the exogenous variable (declared income) to a uniform [0,1] distribution using the empirical distribution function (EDF).
• Model the conditional density Pr(x₀|x₁,…,x_d) as a linear combination of orthonormal polynomials in x₀, with coefficients dependent on endogenous variables via least-squares regression.
• Use coefficients of the polynomial expansion to represent statistical moments (mean, variance, skewness, kurtosis), enabling interpretation similar to cumulants.
• Train the model on 75% of the Polish household budget survey data and evaluate on the remaining 25%, using log-likelihood as the primary metric.
• Apply hierarchical correlation reconstruction (HCR) to model joint densities, with coefficients optimized via mean-square error minimization.
• Use model degree m (up to 9) to control complexity, with m=4 found to balance performance and interpretability.

Experimental results

Research questions

• RQ1 Can hierarchical correlation reconstruction effectively model complex, multimodal conditional densities of income data based on auxiliary variables?
• RQ2 How does the choice of polynomial degree m affect the predictive log-likelihood and credibility assessment accuracy?
• RQ3 To what extent can the coefficients of the polynomial expansion be interpreted as meaningful statistical moments (e.g., mean, variance, skewness)?
• RQ4 Can low predicted density values reliably identify data points that are statistically inconsistent with the sample, indicating potential errors or fraud?
• RQ5 How do higher-order dependencies (e.g., 3-point correlations) improve credibility modeling compared to pairwise-only approaches?

Key findings

• The model achieves a log-likelihood of approximately 1.5 bits higher than the uniform baseline (ρ=1) when using m=2, indicating significantly better prediction of actual income values.
• Model performance peaks at m=4, with a log-likelihood improvement of about 2.0 bits over the uniform baseline, suggesting optimal balance between complexity and accuracy.
• For m=4, the model predicts conditional densities with higher variance than the sample, indicating stronger tail behavior and improved detection of multimodal patterns.
• Approximately 1/3 of predicted distributions have a standard deviation larger than the sample standard deviation, suggesting enhanced sensitivity to extreme values.
• Coefficients for continuous variables (e.g., food, age) show interpretable relationships: negative coefficient for food indicates anticorrelation with income, while positive for age suggests a positive association.
• The method identifies a single 12-person household as an outlier with a surprising spike in predicted income reduction, highlighting sensitivity to rare data points.