[Paper Review] CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training

Qipeng Guo, Zhijing Jin+4 more|arXiv (Cornell University)|Jun 8, 2020

Topic Modeling73 references31 citations

TL;DR

CycleGT jointly learns graph-to-text and text-to-graph generation in an unsupervised cycle-training framework, achieving competitive results with supervised models on WebNLG and strong gains on GenWiki without parallel data.

ABSTRACT

Two important tasks at the intersection of knowledge graphs and natural language processing are graph-to-text (G2T) and text-to-graph (T2G) conversion. Due to the difficulty and high cost of data collection, the supervised data available in the two fields are usually on the magnitude of tens of thousands, for example, 18K in the WebNLG~2017 dataset after preprocessing, which is far fewer than the millions of data for other tasks such as machine translation. Consequently, deep learning models for G2T and T2G suffer largely from scarce training data. We present CycleGT, an unsupervised training method that can bootstrap from fully non-parallel graph and text data, and iteratively back translate between the two forms. Experiments on WebNLG datasets show that our unsupervised model trained on the same number of data achieves performance on par with several fully supervised models. Further experiments on the non-parallel GenWiki dataset verify that our method performs the best among unsupervised baselines. This validates our framework as an effective approach to overcome the data scarcity problem in the fields of G2T and T2G. Our code is available at https://github.com/QipengGuo/CycleGT.

Motivation & Objective

• Address the data scarcity in graph-to-text (G2T) and text-to-graph (T2G) tasks by enabling unsupervised learning from non-parallel graph and text data.
• Develop a cycle-training framework where G2T and T2G act as inverse mappings and improve each other iteratively.
• Demonstrate that CycleGT achieves competitive results with supervised methods on WebNLG 2017/2020 and superior performance on GenWiki under unsupervised settings.

Proposed method

• Formulate G2T as a T5-based sequence-to-sequence model that maps a linearized graph to text.
• Formulate T2G to extract entities with NER, derive entity embeddings, and predict pairwise edge relations via a multi-label classifier.
• Introduce cycle losses that back-translate between graphs and text, creating synthetic paired data: L_CycT and L_CycG.
• Employ iterative back translation, alternately updating G2T and T2G to minimize cycle losses, while handling non-differentiable intermediate outputs.
• Use non-parallel data D_T (text corpus) and D_G (graphs) to train CycleGT without explicit (t,g) pairs.
• Implement with T5 for G2T and a BiLSTM-based T2G, trained for 30 epochs on the WebNLG and GenWiki datasets.

Experimental results

Research questions

• RQ1 Can unsupervised cycle training enable high-quality G2T and T2G generation from non-parallel data?
• RQ2 How does CycleGT perform relative to supervised baselines on standard G2T and T2G benchmarks?
• RQ3 Does CycleGT generalize to large-scale non-parallel data such as GenWiki beyond WebNLG?
• RQ4 What are the practical challenges of cycle training for discrete intermediate outputs, and how can they be addressed?

Key findings

• CycleGT achieves 55.5 BLEU on G2T for WebNLG 2017 in an unsupervised setting, competitive with some supervised baselines.
• CycleGT’s T2G performance on WebNLG 2017 is on par with the G2T component trained with supervised data using the same architecture.
• On GenWiki, CycleGT outperforms unsupervised baselines by a substantial margin (approximately +11 BLEU over the best unsupervised baseline in FINE data and +6 BLEU in FULL data).
• CycleGT demonstrates that unsupervised cycle training can effectively address data scarcity in G2T and T2G, achieving strong results with non-parallel data.