[Paper Review] WikiGraphs: A Wikipedia Text - Knowledge Graph Paired Dataset

Luyu Wang, Yujia Li+2 more|arXiv (Cornell University)|Jul 20, 2021

Topic Modeling4 citations

TL;DR

WikiGraphs introduces a large-scale, automatically constructed dataset pairing full Wikipedia articles with corresponding Freebase knowledge graphs, enabling research in long-form text generation and graph representation learning. The dataset comprises 23,522 graph-text pairs with average article lengths of 3,534 tokens and graphs of 38.7 nodes and 48.3 edges, demonstrating that improved graph conditioning enhances generation and retrieval quality, though significant performance gaps remain for state-of-the-art models.

ABSTRACT

We present a new dataset of Wikipedia articles each paired with a knowledge graph, to facilitate the research in conditional text generation, graph generation and graph representation learning. Existing graph-text paired datasets typically contain small graphs and short text (1 or few sentences), thus limiting the capabilities of the models that can be learned on the data. Our new dataset WikiGraphs is collected by pairing each Wikipedia article from the established WikiText-103 benchmark (Merity et al., 2016) with a subgraph from the Freebase knowledge graph (Bollacker et al., 2008). This makes it easy to benchmark against other state-of-the-art text generative models that are capable of generating long paragraphs of coherent text. Both the graphs and the text data are of significantly larger scale compared to prior graph-text paired datasets. We present baseline graph neural network and transformer model results on our dataset for 3 tasks: graph -> text generation, graph -> text retrieval and text -> graph retrieval. We show that better conditioning on the graph provides gains in generation and retrieval quality but there is still large room for improvement.

Motivation & Objective

• To bridge the gap between long-form text generation models and small-scale graph-text paired datasets by creating a large, scalable, and automatically constructed dataset.
• To enable benchmarking of conditional text generation, graph generation, and graph representation learning on realistic, long-form text and rich knowledge graphs.
• To provide a publicly available resource that supports research in graph-to-text generation, text-to-graph retrieval, and graph-conditioned language modeling.
• To overcome the limitations of prior human-annotated datasets, which are small-scale and restrict model development due to limited graph and text complexity.

Proposed method

• The dataset is constructed by automatically linking each Wikipedia article from the WikiText-103 benchmark to a relevant subgraph from the Freebase knowledge base.
• An automated pipeline extracts a subgraph from Freebase for each Wikipedia article based on entity alignment and semantic relevance.
• The resulting dataset contains 23,522 paired examples, with each Wikipedia article averaging 3,533.8 tokens and each knowledge graph containing 38.7 nodes and 48.3 edges on average.
• The method supports scalability, enabling potential creation of over 3 million graph-text pairs across the entire Wikipedia corpus.
• Baseline models are trained using Transformer-XL architecture, conditioned on knowledge graphs via different encoding strategies: bag-of-words, node features, and graph neural network (GNN) representations.
• The models are evaluated on three tasks: graph-to-text generation, graph-to-text retrieval, and text-to-graph retrieval, using rBLEU as the primary metric.

Experimental results

Research questions

• RQ1 Can graph-conditioned language models generate long, coherent, and relevant text when conditioned on large-scale knowledge graphs?
• RQ2 How does the choice of graph encoding (e.g., bag-of-words, node features, GNN) affect the quality of generated text and retrieval performance?
• RQ3 To what extent does incorporating graph structure improve text generation and retrieval compared to baseline methods?
• RQ4 How do sampling hyperparameters such as length and temperature affect the quality of generated text in graph-conditioned generation?
• RQ5 What is the current performance ceiling of state-of-the-art models on this new large-scale benchmark, and where do major gaps remain?

Key findings

• The GNN-based conditioning method achieved the highest rBLEU score of 36.49 on the validation set for graph-to-text generation with title conditioning, outperforming bag-of-words and node-only baselines.
• Graph-to-text generation performance improved significantly with better graph conditioning, with rBLEU increasing from 9.53 (no conditioning) to 36.49 (GNN with title), indicating strong signal from structured knowledge.
• Longer sample lengths (up to 4096 tokens) led to a decline in performance for all models, suggesting that maintaining topic coherence over long sequences remains a major challenge.
• Temperature hyperparameters had a measurable effect on generation quality, with moderate values (0.8–1.0) yielding better results than very low or high values.
• The performance gap between the best baseline and human reference text remains substantial, indicating significant room for improvement in graph-conditioned generation models.
• The automatic pipeline successfully created a scalable, high-quality dataset with 82.3% coverage of WikiText-103 articles, demonstrating feasibility for large-scale knowledge graph-text pairing.