[Paper Review] Gaussian Process Regression for Binned Data

Michael Thomas Smith, Álvarez+1 more|arXiv (Cornell University)|Sep 6, 2018

Gaussian Processes and Bayesian Inference2 references4 citations

TL;DR

This paper proposes a principled Gaussian Process regression method for binned data by deriving an analytical integral kernel that computes covariance between cuboid regions and points, enabling accurate probabilistic predictions from binned summaries. The method significantly outperforms centroid-based interpolation, especially with few bins, and extends to irregular polytopes via rectangular approximation, improving reconstruction accuracy and enabling applications in differential privacy and integral prediction.

ABSTRACT

Many datasets are in the form of tables of binned data. Performing regression on these data usually involves either reading off bin heights, ignoring data from neighbouring bins or interpolating between bins thus over or underestimating the true bin integrals. In this paper we propose an elegant method for performing Gaussian Process (GP) regression given such binned data, allowing one to make probabilistic predictions of the latent function which produced the binned data. We look at several applications. First, for differentially private regression; second, to make predictions over other integrals; and third when the input regions are irregularly shaped collections of polytopes. In summary, our method provides an effective way of analysing binned data such that one can use more information from the histogram representation, and thus reconstruct a more useful and precise density for making predictions.

Motivation & Objective

• To address the limitation of standard regression on binned data, which often ignores neighboring bin information or distorts bin integrals through interpolation.
• To develop a principled probabilistic method that leverages full histogram information for more accurate latent function reconstruction.
• To extend Gaussian Process regression to non-cuboidal input regions such as census tracts using rectangular approximations.
• To enable differential privacy regression by operating directly on binned data without requiring raw microdata.
• To support prediction of integrals over arbitrary regions, such as age group counts, using binned summaries.

Proposed method

• Derives an analytical integral kernel that computes the covariance between a cuboidal integration region and a point, based on the expectation of the latent function's integral over the region.
• Uses the multivariate normal distribution properties of Gaussian Processes to compute the joint distribution of point values and region integrals.
• Applies the method to binned data by treating bin summaries as known integrals, enabling posterior inference over the latent function.
• For non-cuboidal regions, approximates complex polytopes using sets of hyperrectangles, computing covariance via the integral kernel.
• Implements a point-based approximation method (inspired by Kyriakidis, 2004) as a baseline, then improves it by replacing points with rectangles to reduce approximation error.
• Incorporates non-negativity constraints via Riihimäki and Vehtari (2010) to ensure physically meaningful predictions when modeling count or density data.

Experimental results

Research questions

• RQ1 Can Gaussian Process regression be effectively applied to binned data while respecting the integral constraints of each bin, rather than treating bin means as point observations?
• RQ2 How does the proposed integral kernel method compare to centroid interpolation in terms of predictive accuracy, especially in the presence of sharp peaks or troughs in the underlying function?
• RQ3 Can the method be extended to irregularly shaped input regions such as census tracts, and what approximation strategy yields the best trade-off between accuracy and computational cost?
• RQ4 How does the method perform in differential privacy settings, where raw data is unavailable and only binned summaries are released?
• RQ5 What is the impact of using rectangular vs. point-based approximations for non-rectangular regions in terms of error and computational efficiency?

Key findings

• The integral kernel method significantly outperforms centroid-based interpolation, especially when data is binned into few bins, due to better capture of bin integral constraints.
• On a census-based age-height dataset, the method correctly captures the peak in mid-20s height, while the centroid method underestimates it due to misalignment with bin boundaries.
• For 2D polygons, using rectangular approximation with the integral kernel reduced mean absolute error (MAE) in covariance estimation by up to 60% compared to point-based approximation.
• In 4D hypersphere experiments, the rectangular approximation achieved half the MAE (25.1 vs. 50.0) of the point-based method for the same number of approximating elements.
• The method enables accurate prediction of integrals over regions (e.g., number of young adults), where centroid methods fail by underestimating population in narrow age bands.
• Incorporating non-negativity constraints via Riihimäki and Vehtari (2010) altered the posterior and hyperparameter estimates, though in the tested example, the simpler Gaussian-likelihood model performed slightly better.