[Paper Review] Multi-task Learning for Aggregated Data using Gaussian Processes

Fariba Yousefi, Michael Thomas Smith+1 more|arXiv (Cornell University)|Jun 22, 2019

Gaussian Processes and Bayesian Inference35 references6 citations

TL;DR

This paper proposes a multi-task Gaussian process model for aggregated data that jointly learns variables measured at different spatial or temporal supports using latent functions integrated over varying input scales. By modeling each task as a linear combination of shared latent processes and enabling heterogeneous likelihoods with stochastic variational inference, the framework improves prediction accuracy—demonstrated by a 33% reduction in SMSE when combining low-cost high-frequency sensors with high-accuracy low-frequency data in air pollution modeling.

ABSTRACT

Aggregated data is commonplace in areas such as epidemiology and demography. For example, census data for a population is usually given as averages defined over time periods or spatial resolutions (cities, regions or countries). In this paper, we present a novel multi-task learning model based on Gaussian processes for joint learning of variables that have been aggregated at different input scales. Our model represents each task as the linear combination of the realizations of latent processes that are integrated at a different scale per task. We are then able to compute the cross-covariance between the different tasks either analytically or numerically. We also allow each task to have a potentially different likelihood model and provide a variational lower bound that can be optimised in a stochastic fashion making our model suitable for larger datasets. We show examples of the model in a synthetic example, a fertility dataset, and an air pollution prediction application.

Motivation & Objective

• Address the challenge of modeling multi-scale, aggregated data in fields like epidemiology, demography, and environmental sensing, where observations are collected at different spatial or temporal supports.
• Overcome limitations of existing GP models that assume uniform input supports or homogeneous likelihoods, particularly in handling mixed-type outputs (e.g., continuous, count, binary) across varying data resolutions.
• Develop a scalable and flexible framework that enables joint learning of multiple tasks with different input supports and likelihood models, while preserving cross-task correlation through shared latent processes.
• Enable accurate, high-resolution predictions by leveraging high-frequency, low-accuracy data to inform low-resolution, high-accuracy observations, improving estimation fidelity.

Proposed method

• Model each task as a linear combination of latent Gaussian processes, where the observed task output is the integral of the latent function over a specific support (e.g., time interval or spatial region).
• Use the linear model of coregionalization (LMC) to represent cross-task covariance, allowing for heterogeneous likelihoods (Gaussian, Poisson, etc.) and flexible dependence structures between tasks.
• Derive analytical or numerical expressions for cross-covariance between tasks by integrating the latent GP kernel over their respective supports, enabling exact computation of inter-task correlations.
• Introduce inducing inputs at the latent function level to enable scalable inference via stochastic variational inference (SVI), reducing computational complexity for large datasets.
• Formulate a variational lower bound that supports stochastic optimization using mini-batches, allowing efficient training on large-scale aggregated datasets.
• Apply variational EM to jointly optimize hyperparameters and latent variables, enabling end-to-end learning of the model structure and parameters.

Experimental results

Research questions

• RQ1 How can Gaussian processes be extended to model multi-task learning problems where each task is defined over a different input support (e.g., time period or spatial region) with potentially different likelihoods?
• RQ2 Can the cross-covariance between tasks with different supports be computed analytically or numerically using latent GP integration, and how does this affect predictive performance?
• RQ3 To what extent does incorporating high-frequency, low-accuracy sensor data improve the accuracy of low-resolution, high-accuracy predictions in environmental monitoring applications?
• RQ4 How can scalable inference be achieved in multi-task GP models with heterogeneous likelihoods and multi-scale supports, particularly for large datasets?
• RQ5 Can the proposed framework outperform single-task models or existing multi-task GP approaches in terms of predictive accuracy and robustness when applied to real-world aggregated data?

Key findings

• The model achieved a statistically significant 33% reduction in SMSE (0.439 ± 0.114 vs. 0.657 ± 0.100) when combining high-frequency low-cost sensor data with low-frequency high-accuracy measurements in PM2.5 prediction.
• The inclusion of high-frequency, low-accuracy data improved prediction accuracy for the underlying pollution levels, even when the sensor data was biased, due to the model’s ability to learn from multi-resolution observations.
• The framework successfully preserved the integral constraints of low-resolution data (e.g., 6-hour averages) while producing high-resolution predictions consistent with the observed trends.
• The use of inducing points in the latent space enabled scalable inference, making the model tractable for large datasets through stochastic variational inference.
• The model demonstrated robustness to heterogeneous likelihoods and varying input supports, outperforming single-task baselines in both synthetic and real-world applications.
• The variational EM algorithm with mini-batches enabled efficient optimization, supporting the application of the model to real-world environmental monitoring problems with large-scale data.