[Paper Review] A two-step learning approach for solving full and almost full cold start problems in dyadic prediction

Tapio Pahikkala, Michiel Stock+4 more|arXiv (Cornell University)|May 17, 2014

Tensor decomposition and applications15 references4 citations

TL;DR

This paper proposes a novel two-step learning approach to address full and almost full cold start problems in dyadic prediction, where neither or only one object in a dyad is seen during training. By combining spectral filtering with pairwise feature learning, the method achieves slightly better predictive performance than tensor product kernel methods, with closed-form solutions enabling efficient cross-validation and implementation in both cold start settings.

ABSTRACT

Dyadic prediction methods operate on pairs of objects (dyads), aiming to infer labels for out-of-sample dyads. We consider the full and almost full cold start problem in dyadic prediction, a setting that occurs when both objects in an out-of-sample dyad have not been observed during training, or if one of them has been observed, but very few times. A popular approach for addressing this problem is to train a model that makes predictions based on a pairwise feature representation of the dyads, or, in case of kernel methods, based on a tensor product pairwise kernel. As an alternative to such a kernel approach, we introduce a novel two-step learning algorithm that borrows ideas from the fields of pairwise learning and spectral filtering. We show theoretically that the two-step method is very closely related to the tensor product kernel approach, and experimentally that it yields a slightly better predictive performance. Moreover, unlike existing tensor product kernel methods, the two-step method allows closed-form solutions for training and parameter selection via cross-validation estimates both in the full and almost full cold start settings, making the approach much more efficient and straightforward to implement.

Motivation & Objective

• Address the full and almost full cold start problem in dyadic prediction, where both or one object in a dyad is unseen during training.
• Overcome limitations of existing kernel-based approaches, which lack closed-form solutions for training and cross-validation in cold start settings.
• Develop a method that enables efficient, scalable, and theoretically grounded prediction when training data is sparse or absent for one or both objects.
• Improve predictive performance over existing pairwise learning and kernel methods in low-data regimes.

Proposed method

• Propose a two-step learning algorithm that first learns representations for individual objects (e.g., drugs, targets) using spectral filtering, then combines them into dyadic predictions.
• Use tensor product feature representations of dyads as input, enabling the model to capture interactions between object pairs.
• Formulate the learning process such that both model training and hyperparameter selection via cross-validation admit closed-form solutions in both full and almost full cold start settings.
• Theoretical analysis shows strong mathematical equivalence between the two-step method and tensor product kernel methods, justifying its design.
• Apply regularization independently to each object type (e.g., drugs and targets), enhancing generalization in low-data regimes.
• Implement the method using kernel ridge regression or support vector machines with the learned two-step features, enabling efficient inference.

Experimental results

Research questions

• RQ1 Can a two-step learning approach outperform traditional tensor product kernel methods in full and almost full cold start dyadic prediction?
• RQ2 Does the proposed method allow closed-form solutions for both model training and cross-validation in cold start settings?
• RQ3 How does the performance of the two-step method compare to kernel-based and single-task learning in low-data regimes?
• RQ4 Under what data conditions does auxiliary information from related tasks provide positive transfer in cold start prediction?

Key findings

• The two-step method achieves slightly better predictive performance than pairwise kernel ridge regression (KRR) in full and almost full cold start problems.
• Unlike existing kernel methods, the two-step approach enables closed-form solutions for both model training and cross-validation, significantly improving computational efficiency.
• In experiments on UniProt and 20 Newsgroups data, the two-step method outperforms pairwise KRR and performs competitively with single-task learning when target data is limited.
• Positive transfer from auxiliary tasks is observed only when target data is scarce; once sufficient target data is available, auxiliary data no longer improves performance.
• The method remains effective even with as few as 50 training proteins, and performance peaks when auxiliary and target data are balanced, though it eventually falls below single-task learning at high data volumes.
• The two-step approach is robust across diverse domains, including bioinformatics, chemistry, and text similarity, demonstrating broad applicability.