[Paper Review] A Unifying View of Multiple Kernel Learning

Marius Kloft, Ulrich Rückert+1 more|arXiv (Cornell University)|May 4, 2010

Sparse and Compressive Sensing Techniques14 references21 citations

TL;DR

This paper presents a unifying optimization framework for multiple kernel learning (MKL) that subsumes existing approaches—such as $β$-norm MKL, elastic net, and block-norm MKL—under a single criterion with a dual-block-norm regularizer. It derives a modular dual representation enabling efficient smooth optimization and empirically shows that $∞$-norm MKL outperforms other variants in bioinformatics and intrusion detection tasks, with the best generalization bound matching prior state-of-the-art.

ABSTRACT

Recent research on multiple kernel learning has lead to a number of approaches for combining kernels in regularized risk minimization. The proposed approaches include different formulations of objectives and varying regularization strategies. In this paper we present a unifying general optimization criterion for multiple kernel learning and show how existing formulations are subsumed as special cases. We also derive the criterion's dual representation, which is suitable for general smooth optimization algorithms. Finally, we evaluate multiple kernel learning in this framework analytically using a Rademacher complexity bound on the generalization error and empirically in a set of experiments.

Motivation & Objective

• To unify diverse multiple kernel learning (MKL) formulations into a single, general optimization criterion.
• To derive a dual representation of the MKL criterion that enables flexible and efficient optimization using standard smooth solvers.
• To analyze generalization performance via a new Rademacher complexity concentration bound applicable to elastic net and block-norm MKL.
• To empirically evaluate and compare MKL variants across bioinformatics and intrusion detection datasets.
• To determine whether non-sparse $∞$-norm MKL outperforms sparse and other block-norm variants in practice.

Proposed method

• Formulate MKL as a regularized risk minimization problem with a dual-block-norm regularizer, generalizing $β$-norm, elastic net, and block-norm approaches.
• Derive a modular dual representation that separates loss and regularizer contributions, enabling plug-and-play use of different convex loss functions.
• Use smooth optimization techniques to solve the dual problem efficiently, leveraging the separability of the dual objective.
• Apply the framework to real-world datasets: promoter prediction (yeast) and network intrusion detection (HTTP traffic).
• Implement and compare multiple MKL variants: $β$-norm MKL, elastic net with varying $μ$, and block-norm MKL with $p \in \{1, 4/3, 2, 4, \infty\}$.
• Use Rademacher complexity to derive a generalization bound that matches the best-known bound for $β=1$ and $β=4/3$.

Experimental results

Research questions

• RQ1 Can all major MKL formulations—$β$-norm, elastic net, and block-norm—be unified under a single optimization criterion?
• RQ2 Does the proposed dual-block-norm regularizer enable a modular and flexible dual optimization framework?
• RQ3 What is the generalization performance of the unified MKL framework, and how does it compare to prior bounds?
• RQ4 Does the $∞$-norm MKL formulation outperform other MKL variants in terms of predictive accuracy on real-world data?
• RQ5 How does the choice of regularization (e.g., $β$-norm, elastic net) affect model sparsity and performance in practice?

Key findings

• The $∞$-norm MKL achieved the highest prediction accuracy of 87.67±0.09 on the promoter prediction dataset, outperforming both $β=1$ and $β=4/3$ block-norm MKL.
• On the intrusion detection dataset, $∞$-norm MKL achieved an AUC [0,0.1] of 99.68±0.09, the highest among all tested models.
• The elastic net with $μ=0.1$ achieved 99.46±0.13 AUC [0,0.1], slightly outperforming $β=1$-norm MKL and matching the performance of $β=4/3$-norm MKL.
• The generalization bound derived via Rademacher complexity matches the best-known bound for $β=1$ and $β=4/3$, validating the theoretical robustness of the framework.
• Empirically, $∞$-norm MKL consistently outperformed other block-norm variants across both datasets, despite being non-sparse.
• The framework successfully unified diverse MKL approaches under a single criterion, enabling a systematic comparison of regularization strategies.