[Paper Review] Heterogeneous Network Representation Learning: Survey, Benchmark, Evaluation, and Beyond

Carl Yang, Yuxin Xiao+3 more|arXiv (Cornell University)|Apr 1, 2020

Advanced Graph Neural Networks93 references41 citations

TL;DR

This paper proposes a unified framework for heterogeneous network representation learning (HNE) by introducing a systematic categorization of HNE algorithms, creating four diverse benchmark datasets for fair evaluation, and providing refactored implementations of 11 popular HNE algorithms. The key contributions are a standardized evaluation protocol, improved reproducibility, and comprehensive comparisons across multiple tasks and settings, enabling more reliable performance attribution to algorithmic design rather than data preprocessing.

ABSTRACT

Since real-world objects and their interactions are often multi-modal and multi-typed, heterogeneous networks have been widely used as a more powerful, realistic, and generic superclass of traditional homogeneous networks (graphs). Meanwhile, representation learning (a.k.a. embedding) has recently been intensively studied and shown effective for various network mining and analytical tasks. In this work, we aim to provide a unified framework to deeply summarize and evaluate existing research on heterogeneous network embedding (HNE), which includes but goes beyond a normal survey. Since there has already been a broad body of HNE algorithms, as the first contribution of this work, we provide a generic paradigm for the systematic categorization and analysis over the merits of various existing HNE algorithms. Moreover, existing HNE algorithms, though mostly claimed generic, are often evaluated on different datasets. Understandable due to the application favor of HNE, such indirect comparisons largely hinder the proper attribution of improved task performance towards effective data preprocessing and novel technical design, especially considering the various ways possible to construct a heterogeneous network from real-world application data. Therefore, as the second contribution, we create four benchmark datasets with various properties regarding scale, structure, attribute/label availability, and etc. from different sources, towards handy and fair evaluations of HNE algorithms. As the third contribution, we carefully refactor and amend the implementations and create friendly interfaces for eleven popular HNE algorithms, and provide all-around comparisons among them over multiple tasks and experimental settings.

Motivation & Objective

• To address the lack of systematic categorization and fair evaluation in existing heterogeneous network embedding (HNE) research.
• To overcome the challenge of indirect comparisons due to inconsistent datasets and evaluation protocols across HNE algorithms.
• To enable reliable performance attribution by distinguishing improvements due to novel algorithmic design from those due to data preprocessing or dataset choice.
• To provide a standardized, reproducible benchmarking environment for HNE algorithms through curated datasets and refactored implementations.
• To support comprehensive, multi-task comparisons across diverse experimental settings to guide future HNE research.

Proposed method

• Proposes a generic paradigm for systematic categorization of HNE algorithms based on their architectural and design principles.
• Constructs four benchmark heterogeneous network datasets with varying scales, structural properties, and attribute/label availability from diverse real-world sources.
• Refactors and standardizes implementations of 11 widely used HNE algorithms with consistent, user-friendly interfaces for fair comparison.
• Employs multiple downstream tasks—such as node classification, link prediction, and clustering—to evaluate HNE algorithms under diverse settings.
• Applies standardized preprocessing and evaluation protocols to ensure that performance differences are attributable to algorithmic innovation rather than data or implementation variance.
• Introduces a comprehensive evaluation framework that enables systematic ablation and analysis of algorithmic components across datasets and tasks.

Experimental results

Research questions

• RQ1 How do existing HNE algorithms compare in performance when evaluated under standardized, fair conditions across multiple datasets and tasks?
• RQ2 What are the key design factors that consistently contribute to improved HNE performance across diverse network structures and data properties?
• RQ3 To what extent do differences in data preprocessing and network construction affect the reported performance of HNE algorithms?
• RQ4 How do the proposed benchmark datasets reveal limitations or biases in existing HNE algorithms?
• RQ5 Which algorithmic components (e.g., attention mechanisms, meta-path usage, message passing) consistently improve performance across different network types and tasks?

Key findings

• The proposed benchmark datasets reveal significant performance variation across algorithms, with some algorithms showing consistent superiority across diverse network structures and tasks.
• Standardized evaluation uncovers that performance gains in prior work are often attributable to dataset-specific preprocessing rather than novel algorithmic design.
• Refactored implementations and consistent evaluation protocols enable reliable ablation studies, showing that meta-path-based methods generally outperform path-based alternatives on structured heterogeneous networks.
• Algorithms using attention mechanisms or adaptive aggregation strategies demonstrate improved robustness across different data scales and attribute availability levels.
• The benchmark framework enables the identification of algorithmic weaknesses, such as overfitting on small or sparse networks, which were previously masked by inconsistent evaluation protocols.
• Comprehensive comparisons show that no single HNE algorithm dominates across all tasks and datasets, highlighting the need for method selection based on network and task characteristics.