[Paper Review] Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment

Dinghan Shen, Xinyuan Zhang+2 more|arXiv (Cornell University)|Aug 29, 2018

Advanced Graph Neural Networks27 references4 citations

TL;DR

This paper proposes WANE-ww, a novel network embedding framework that enhances semantic-aware representations by performing fine-grained, word-by-word alignment between text sequences associated with vertex pairs. By using element-wise subtraction for matching and max-pooling for aggregation, the model captures meaningful semantic relationships, achieving state-of-the-art performance on link prediction and vertex classification across three real-world datasets, with gains up to 85.1% accuracy on Cora at 10% training ratio.

ABSTRACT

Network embeddings, which learn low-dimensional representations for each vertex in a large-scale network, have received considerable attention in recent years. For a wide range of applications, vertices in a network are typically accompanied by rich textual information such as user profiles, paper abstracts, etc. We propose to incorporate semantic features into network embeddings by matching important words between text sequences for all pairs of vertices. We introduce a word-by-word alignment framework that measures the compatibility of embeddings between word pairs, and then adaptively accumulates these alignment features with a simple yet effective aggregation function. In experiments, we evaluate the proposed framework on three real-world benchmarks for downstream tasks, including link prediction and multi-label vertex classification. Results demonstrate that our model outperforms state-of-the-art network embedding methods by a large margin.

Motivation & Objective

• To address the limitation of existing network embedding methods that treat text sequences as whole embeddings, ignoring fine-grained semantic word-level relationships.
• To develop a framework that explicitly models word-by-word compatibility between text sequences of vertex pairs to improve semantic-aware network representations.
• To create an efficient, scalable method that avoids complex RNN/CNN architectures while maintaining high performance on downstream tasks.
• To demonstrate that adaptive, context-aware word alignment leads to more informative and robust network embeddings compared to existing baselines.

Proposed method

• The model introduces a word-by-word alignment mechanism that computes matching vectors between each word in one sentence and key words from the other via an attention-based weighting scheme.
• Matching features are computed using element-wise operations—subtraction, multiplication, or their concatenation—on aligned word embeddings to capture semantic compatibility.
• A max-pooling aggregation function is applied to select the most salient alignment features across all word pairs, producing a compact sentence-level representation.
• The final network embedding is learned by jointly optimizing structural information from the graph and the aligned semantic features from text sequences.
• The framework is designed to be highly parallelizable and does not require recurrent or convolutional encoders, enabling faster training on large-scale networks.
• The model is trained end-to-end using a contrastive loss objective to preserve structural and semantic similarity between connected vertices.

Experimental results

Research questions

• RQ1 Can fine-grained word-by-word alignment between text sequences improve the quality of network embeddings compared to sentence-level matching?
• RQ2 How does the choice of matching operation (e.g., subtraction, multiplication) affect performance on downstream tasks?
• RQ3 Does max-pooling aggregation outperform mean- or CNN-based pooling in capturing relevant semantic features for vertex representation?
• RQ4 To what extent does the proposed method improve performance on link prediction and semi-supervised vertex classification compared to state-of-the-art baselines?
• RQ5 Can the model highlight meaningful word pairs (e.g., keywords) that explain the predicted relationships between vertices?

Key findings

• On the Cora dataset, WANE-ww achieved 85.1% accuracy in semi-supervised vertex classification with only 10% labeled training data, outperforming the next-best baseline (CANE) by 1.7 percentage points.
• On the HepTh dataset, WANE-ww achieved 82.7% accuracy at 7% training ratio, surpassing the previous state-of-the-art (CANE) by 2.3 percentage points.
• Element-wise subtraction was found to be the most effective matching operation across all three datasets, consistently outperforming multiplication and their concatenation.
• Max-pooling aggregation yielded the best results on all datasets, indicating its effectiveness in selecting the most informative word-level alignment features.
• t-SNE visualization confirmed that WANE-ww embeddings cluster by label, even without label supervision, demonstrating meaningful semantic structure learning.
• Case studies revealed that key technical terms like 'MCMC' and 'convergence' were assigned high matching vector norms, confirming the model's ability to identify semantically relevant word pairs.