[Paper Review] Local Gaussian Regression

Franziska Meier, Philipp Hennig+1 more|arXiv (Cornell University)|Feb 4, 2014

Gaussian Processes and Bayesian Inference17 references3 citations

TL;DR

This paper proposes Local Gaussian Regression (LGR), a probabilistic, locally weighted regression method that combines the computational efficiency of locally weighted regression (LWR) with the generative modeling capabilities of Gaussian process regression. By using localized feature functions and a variational approximation, LGR achieves linear computational complexity while enabling collaborative learning across local models, outperforming LWR and mixture of experts in accuracy and robustness to hyperparameter tuning.

ABSTRACT

Locally weighted regression was created as a nonparametric learning method that is computationally efficient, can learn from very large amounts of data and add data incrementally. An interesting feature of locally weighted regression is that it can work with spatially varying length scales, a beneficial property, for instance, in control problems. However, it does not provide a generative model for function values and requires training and test data to be generated identically, independently. Gaussian (process) regression, on the other hand, provides a fully generative model without significant formal requirements on the distribution of training data, but has much higher computational cost and usually works with one global scale per input dimension. Using a localising function basis and approximate inference techniques, we take Gaussian (process) regression to increasingly localised properties and toward the same computational complexity class as locally weighted regression.

Motivation & Objective

• To develop a probabilistic regression model that retains the computational efficiency of locally weighted regression (LWR) while enabling generative modeling and collaborative learning across local models.
• To address the hyperparameter sensitivity and lack of generative structure in LWR by embedding local models within a Gaussian process framework.
• To reduce the computational cost of Gaussian process regression through a variational approximation that decouples local models, achieving linear complexity in the number of local models.
• To enable incremental learning and robust performance across diverse regression tasks, especially in high-dimensional and real-time settings like robot control.

Proposed method

• Represents local models using localized feature functions—specifically, radial basis functions—within a generalized linear regression framework to encode spatial localization.
• Reformulates Gaussian process regression as a probabilistic model with local feature functions, enabling a fully generative model for function values.
• Applies a variational inference approximation that assumes local models are approximately independent due to spatial localization, reducing inference cost from cubic to linear in the number of local models.
• Uses maximum likelihood estimation to learn model parameters, including length scales and regularization, with a trade-off parameter $ w_{\text{gen}} $ controlling model complexity and sparsity.
• Employs a pruning mechanism that automatically reduces the number of active local models based on their contribution, improving efficiency without sacrificing accuracy.
• Extends the framework to a local nonparametric formulation by allowing flexible, data-driven local feature representations.

Experimental results

Research questions

• RQ1 Can a probabilistic regression model be designed to achieve the computational efficiency of locally weighted regression while retaining a fully generative model for function values?
• RQ2 How can local models be made to collaborate during training without incurring the full computational cost of global Gaussian process inference?
• RQ3 To what extent does the proposed method reduce hyperparameter sensitivity compared to LWR and mixture of experts?
• RQ4 Can the model maintain high accuracy with fewer local models and less tuning, especially in high-dimensional or real-time settings?

Key findings

• LGR outperforms both LWR and mixture of experts (ME) in predictive accuracy on the cross function benchmark, achieving lower normalized mean squared error with fewer local models.
• The performance of LGR is robust to the choice of the $ w_{\text{gen}} $ hyperparameter, with only a slight improvement in accuracy for higher values, indicating low sensitivity to tuning.
• In the SARCOS inverse dynamics task, LGR achieved superior predictive accuracy across all 7 joints compared to LWR and ME, demonstrating strong generalization in high-dimensional settings.
• Despite allowing up to 2000 local models for $ w_{\text{gen}} = 1 $, LGR’s pruning mechanism limited the final number of active models to a small fraction, showing effective model compression.
• The variational approximation successfully decoupled local models, reducing computational cost to linear complexity while preserving model quality.
• LGR’s generative formulation enables principled learning of parameters and model structure, unlike LWR, which treats training and test data differently.