[Paper Review] Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks

Tianyu Fan, Lirong Wu+5 more|ArXiv.org|Mar 3, 2024

Multi-Criteria Decision MakingDecision Sciences3 citations

TL;DR

This paper proposes WAS (Weigh and Select), a novel instance-level framework that decouples task selection and weighting for integrating multiple graph pre-training tasks. By using decoupled siamese networks, WAS adaptively learns optimal task combinations and instance-specific weights, achieving state-of-the-art performance on 16 graph datasets with consistent gains as the task pool expands.

ABSTRACT

Recent years have witnessed the great success of graph pre-training for graph representation learning. With hundreds of graph pre-training tasks proposed, integrating knowledge acquired from multiple pre-training tasks has become a popular research topic. In this paper, we identify two important collaborative processes for this topic: (1) select: how to select an optimal task combination from a given task pool based on their compatibility, and (2) weigh: how to weigh the selected tasks based on their importance. While there currently has been a lot of work focused on weighing, comparatively little effort has been devoted to selecting. This paper proposes a novel instance-level framework for integrating multiple graph pre-training tasks, Weigh And Select (WAS), where the two collaborative processes, weighing and selecting, are combined by decoupled siamese networks. Specifically, it first adaptively learns an optimal combination of tasks for each instance from a given task pool, based on which a customized instance-level task weighing strategy is learned. Extensive experiments on 16 graph datasets across node-level and graph-level downstream tasks have demonstrated that by combining a few simple but classical tasks, WAS can achieve comparable performance to other leading counterparts. The code is available at https://github.com/TianyuFan0504/WAS.

Motivation & Objective

• Address the gap in existing methods that focus only on task weighting while neglecting task compatibility and selection.
• Investigate the necessity of decoupling task selection (compatibility) from task weighting (importance) in multi-task graph pre-training.
• Develop a unified framework that customizes both task selection and weighting per downstream instance to improve representation learning.
• Demonstrate the effectiveness of the proposed method on both node-level and graph-level downstream tasks, including under-explored graph-level benchmarks.
• Show that performance improves consistently as the task pool expands, unlike prior methods that plateau or degrade.

Proposed method

• Propose a decoupled siamese network architecture to separately model task selection and task weighting processes.
• Use a selection head with a learnable projection (MLP) to generate instance-specific task combination probabilities.
• Use a weighting head to predict instance-level task importance weights, ensuring they sum to one via re-weighting.
• Train the framework end-to-end using a contrastive learning objective on graph representations.
• Integrate multiple classical pre-training tasks (e.g., AttrMask, ContextPred, EdgePred, GraphCL) in a unified, adaptive manner.
• Apply the framework during the pre-training stage to enable instance-aware, dynamic task combination and weighting.

Experimental results

Research questions

• RQ1 Can decoupling task selection and weighting improve performance in multi-task graph pre-training?
• RQ2 How does task compatibility affect performance when combining multiple pre-training tasks, and can it be effectively modeled?
• RQ3 Does instance-level customization of task selection and weighting lead to better downstream performance than global or fixed strategies?
• RQ4 How does the performance of the method scale with an increasing number of pre-training tasks in the pool?
• RQ5 What is the relative contribution of selection versus weighting in achieving high performance?

Key findings

• WAS achieves state-of-the-art performance on 16 graph datasets, with a mean ROC-AUC of 80.7% on BACE, outperforming all baselines.
• Performance improves consistently as the task pool expands, with WAS showing gains even with 10+ tasks, unlike AGSSL which plateaus.
• The ablation study confirms that importance-based selection performs worse than random selection, proving that compatibility must be addressed separately.
• Removing the projection head (MLP) in the selection module degrades performance, highlighting its role in decoupling selection from weighting.
• Re-weighing after selection is critical—without it, the sum of weights is not normalized, leading to performance degradation.
• Visualization shows that different instances select different task combinations (e.g., IG, AM, GraphLoG), confirming the model’s ability to learn instance-specific strategies.