[Paper Review] Fast multi-output relevance vector regression

Youngmin Ha|arXiv (Cornell University)|Apr 17, 2017

Face and Expression Recognition23 references4 citations

TL;DR

This paper proposes a faster multi-output relevance vector regression (MRVR) algorithm by modeling correlated outputs using a matrix normal distribution instead of a multivariate normal distribution, reducing time complexity from O(VM³) to O(V³ + M³). The method achieves significantly faster computation—especially for larger output dimensions—while improving noise covariance estimation, though at the cost of slightly reduced mean prediction accuracy and increased relevance vector counts due to a structural assumption on weight regularization.

ABSTRACT

This paper aims to decrease the time complexity of multi-output relevance vector regression from O(VM^3) to O(V^3+M^3), where V is the number of output dimensions, M is the number of basis functions, and V

Motivation & Objective

• To reduce the computational time complexity of multi-output relevance vector regression (MRVR) for large-scale, multi-dimensional output problems.
• To improve the accuracy of noise covariance matrix estimation by modeling output correlations explicitly through a matrix normal distribution.
• To address the inefficiency of existing MRVR methods that scale poorly with increasing output dimensions (V) and basis functions (M).
• To balance computational speed with predictive accuracy, identifying trade-offs introduced by structural assumptions in the model.

Proposed method

• Replaces the multivariate normal likelihood model in traditional MRVR with a matrix normal distribution to better capture cross-output correlations in the noise structure.
• Derives a new closed-form solution for the posterior distribution of weights using matrix calculus, enabling faster computation via efficient matrix inversions.
• Introduces a key assumption Ω = E[WᵀW]/tr(A⁻¹) to constrain the weight matrix, which accelerates computation but affects mean prediction accuracy.
• Employs the kernel trick and sparse Bayesian learning principles to maintain model flexibility while reducing computational burden.
• Uses matrix inversion operations on V×V and M×M matrices (O(V³ + M³)) instead of the original O(VM³) complexity, leveraging the condition V < M in most applications.
• Implements the algorithm in MATLAB with publicly available code to ensure reproducibility and facilitate adoption.

Experimental results

Research questions

• RQ1 Can the time complexity of multi-output relevance vector regression be reduced from O(VM³) to O(V³ + M³) without sacrificing predictive performance?
• RQ2 How does modeling output correlations via a matrix normal distribution improve noise covariance estimation compared to independent Gaussian assumptions?
• RQ3 What trade-offs arise from the structural assumption Ω = E[WᵀW]/tr(A⁻¹) in terms of mean prediction accuracy and relevance vector count?
• RQ4 How does the proposed method scale with increasing numbers of output dimensions (V) and basis functions (M)?

Key findings

• The proposed MRVR method reduces time complexity from O(VM³) to O(V³ + M³), resulting in significantly faster computation, especially when V and M are large.
• The method achieves higher accuracy in estimating the noise covariance matrix Ω, as evidenced by lower RMSE in covariance estimation across all tested configurations.
• Computation time differences between the proposed and existing methods increase with larger V and N, confirming the theoretical complexity advantage.
• The proposed method exhibits higher RMSE in mean prediction (Table 5), particularly for high V and low N, indicating reduced accuracy in estimating the true function mean.
• The number of relevance vectors (RVs) is higher in the proposed method (Table 6), suggesting a less sparse solution due to the weight regularization assumption.
• Despite trade-offs in mean accuracy and sparsity, the method is more efficient and accurate in noise covariance estimation, making it suitable for applications where these are prioritized.