[Paper Review] Knowledge Graph Embedding with Atrous Convolution and Residual Learning

Feiliang Ren, Juchen Li+5 more|arXiv (Cornell University)|Oct 23, 2020

Advanced Graph Neural Networks51 references4 citations

TL;DR

This paper proposes AcrE, a knowledge graph embedding model that leverages atrous convolution and residual learning to enhance feature interaction and mitigate gradient vanishing while maintaining high parameter efficiency. It achieves state-of-the-art performance on six benchmark datasets with fewer parameters than competing methods.

ABSTRACT

Knowledge graph embedding is an important task and it will benefit lots of downstream applications. Currently, deep neural networks based methods achieve state-of-the-art performance. However, most of these existing methods are very complex and need much time for training and inference. To address this issue, we propose a simple but effective atrous convolution based knowledge graph embedding method. Compared with existing state-of-the-art methods, our method has following main characteristics. First, it effectively increases feature interactions by using atrous convolutions. Second, to address the original information forgotten issue and vanishing/exploding gradient issue, it uses the residual learning method. Third, it has simpler structure but much higher parameter efficiency. We evaluate our method on six benchmark datasets with different evaluation metrics. Extensive experiments show that our model is very effective. On these diverse datasets, it achieves better results than the compared state-of-the-art methods on most of evaluation metrics. The source codes of our model could be found at https://github.com/neukg/AcrE.

Motivation & Objective

• To address the limitations of deep neural network-based knowledge graph embedding methods, including high complexity and reduced feature resolution due to striding in convolutional layers.
• To mitigate the vanishing/exploding gradient problem and original information forgetting in deep networks through residual learning.
• To improve feature interaction in knowledge graph embeddings using atrous convolutions without increasing model parameters significantly.
• To achieve higher parameter efficiency while maintaining or improving performance compared to state-of-the-art models.
• To propose a simple yet effective architecture combining atrous convolutions and residual connections for knowledge graph embedding.

Proposed method

• The model employs atrous convolution to expand the感受 field of filters without increasing parameters or computational cost.
• Residual learning is integrated to preserve original features and stabilize training, reducing information loss in deep architectures.
• Two integration structures are proposed: a serial structure and a parallel structure, combining multiple convolutional operations.
• The serial structure applies convolutions sequentially with skip connections, while the parallel structure uses element-wise addition or concatenation to fuse features from different convolutions.
• Atrous rates are used to control感受 field expansion, allowing effective context aggregation across relations and entities.
• The model is trained using a margin-based ranking loss function, similar to previous KGE methods, to optimize entity and relation representations.

Experimental results

Research questions

• RQ1 Can atrous convolution effectively enhance feature interaction in knowledge graph embeddings without increasing model complexity?
• RQ2 Does residual learning significantly improve training stability and performance in deep KGE models by mitigating information forgetting and gradient issues?
• RQ3 How does the proposed AcrE model compare in performance and parameter efficiency to state-of-the-art KGE models across diverse benchmark datasets?
• RQ4 What is the impact of different integration strategies (element-wise addition vs. concatenation) on model performance in the parallel structure?
• RQ5 Can the model maintain high efficiency in training and inference while outperforming complex DNN-based baselines?

Key findings

• AcrE achieves state-of-the-art performance on six benchmark datasets, outperforming existing methods on most evaluation metrics.
• The serial residual structure achieves greater performance gains from residual learning than the parallel structure, due to higher information forgetting in deeper sequential layers.
• The concatenation-based integration method consistently outperforms the element-wise addition method in the parallel structure.
• AcrE has significantly better parameter efficiency than most baselines, with only ~5.6 million parameters in the serial version and ~6.2 million in the parallel version, compared to up to 29.79 million in HAKE.
• Training time is substantially faster than many recent GNN and DNN-based models, taking only ~100 seconds per epoch on FB15k-237 with a Titan XP GPU, and inference takes just 6 seconds for the full test set.
• The model demonstrates strong generalization across diverse datasets, confirming its robustness and effectiveness in real-world knowledge graph scenarios.