[Paper Review] Multiple Kernel Learning: A Unifying Probabilistic Viewpoint

Hannes Nickisch, Matthias Seeger|arXiv (Cornell University)|Mar 4, 2011

Gaussian Processes and Bayesian Inference16 references8 citations

TL;DR

This paper unifies multiple kernel learning (MKL) under a probabilistic framework, showing that regularized risk methods are special cases of approximate Bayesian inference. It proposes a provably convergent, efficient double-loop optimization algorithm that jointly optimizes kernel weights and latent functions, applicable to regression, robust regression, and classification.

ABSTRACT

We present a probabilistic viewpoint to multiple kernel learning unifying well-known regularised risk approaches and recent advances in approximate Bayesian inference relaxations. The framework proposes a general objective function suitable for regression, robust regression and classification that is lower bound of the marginal likelihood and contains many regularised risk approaches as special cases. Furthermore, we derive an efficient and provably convergent optimisation algorithm.

Motivation & Objective

• To unify regularized risk approaches and probabilistic kernel learning under a common theoretical framework.
• To clarify the connections between widely used but disparate MKL methods from the regularized risk and Bayesian inference camps.
• To develop a general-purpose optimization algorithm that efficiently handles multiple kernel learning across different learning tasks.
• To provide a unified view of kernel learning that subsumes MAP, marginal likelihood maximization, and variational Bayes approximations.
• To ensure convergence and computational efficiency without requiring external optimization libraries.

Proposed method

• Proposes a unifying objective function that serves as a lower bound on the marginal likelihood and generalizes regularized risk criteria.
• Derives a double-loop optimization algorithm alternating between updating latent function values and kernel weights via Newton-type steps.
• Uses Legendre duality to handle the log-determinant term in the marginal likelihood, enabling convex relaxation of the non-convex optimization problem.
• Employs a barrier method to enforce non-negativity constraints on kernel weights, ensuring numerical stability.
• Applies a tangential approximation to the log-determinant term, allowing efficient Hessian computation and provable convergence.
• Adapts the framework to variational Bayes and MAP estimation by reparametrizing latent variables and using Cholesky factorizations for efficient matrix operations.

Experimental results

Research questions

• RQ1 How can regularized risk minimization and probabilistic kernel learning be formally connected within a single framework?
• RQ2 What common optimization structure underlies diverse MKL approaches, including MAP, MLM, and variational Bayes?
• RQ3 Can a single, efficient, and provably convergent algorithm be designed for multiple kernel learning across regression, robust regression, and classification?
• RQ4 What is the role of the log-determinant term in the marginal likelihood, and how can it be efficiently approximated in non-convex settings?
• RQ5 How do the algorithmic structures of MAP, MLM, and VB relate, and what insights emerge from their unified treatment?

Key findings

• The proposed framework unifies regularized risk, MAP, and marginal likelihood maximization as special cases of a single probabilistic objective function.
• The double-loop algorithm achieves provable convergence by leveraging tangential approximation and Newton steps, with per-iteration cost of $ O(n^3 + M n^2) $.
• The method efficiently handles non-convexity through a barrier method and dual parameterization, ensuring kernel weights remain non-negative.
• For variational Bayes, the algorithm requires $ O(M n^3) $ cost due to Cholesky factorizations, matching complexity of standard Laplace approximations.
• The framework reveals that joint MAP and VB share the same convexity structure when reparametrized, enabling unified optimization.
• Empirical validation shows the algorithm converges faster than standard alternating minimization, with improved stability and accuracy across regression and classification tasks.