[Paper Review] Evaluating Generative Models for Graph-to-Text Generation

Shuzhou Yuan, Michael Färber|arXiv (Cornell University)|Jul 27, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper evaluates zero-shot graph-to-text generation using GPT-3 and ChatGPT on AGENDA and WebNLG datasets, comparing their performance to fine-tuned models like T5 and BART. Despite generating fluent text with BLEU scores of 10.57 and 11.08, generative models struggle with semantic relation understanding and produce hallucinations, while BERT fine-tuned on generated text achieves over 95% macro-F1 in distinguishing machine-generated from human-written text.

ABSTRACT

Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative models are capable of generating fluent and coherent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to detect machine-generated text and achieve high macro-F1 scores. We have made the text generated by generative models publicly available.

Motivation & Objective

• To assess the zero-shot capability of generative models like GPT-3 and ChatGPT in generating descriptive text from structured graph data.
• To compare the performance of these generative models against fine-tuned sequence-to-sequence models such as T5 and BART on standard graph-to-text benchmarks.
• To analyze failure modes in generated text, particularly regarding semantic relation understanding and hallucination.
• To investigate the detectability of machine-generated text using fine-tuned BERT for binary classification.
• To provide publicly available generated text for future research on trustworthy AI and machine-generated text analysis.

Proposed method

• Linearized graph representations are used as input prompts for GPT-3 and ChatGPT, transforming structured knowledge into natural language sequences.
• The models are prompted with instruction templates to generate text from graph triples, enabling zero-shot inference without fine-tuning.
• Performance is evaluated using standard metrics: BLEU, METEOR, and ROUGE on test sets from AGENDA and WebNLG.
• Error analysis is conducted by comparing generated outputs with reference texts to identify hallucinations and relation comprehension failures.
• A BERT model is fine-tuned on paired human-written and machine-generated text to classify the source of text, with evaluation on macro-F1 scores.
• All generated text is released publicly via GitHub to support reproducibility and future analysis of model-generated content.

(a) Generate paper abstract from title, entities and graph: <title> Significance-aware Hammerstein group models for nonlinear acoustic echo cancellation. <entities> non-linear preprocessor echo path hammerstein model <graph> <H> non-linear preprocessor <R> USED-FOR <T> echo path <H> preprocessor <R>

Experimental results

Research questions

• RQ1 Can GPT-3 and ChatGPT generate fluent and coherent text from graph-structured data in a zero-shot setting?
• RQ2 How do the performance metrics (BLEU, METEOR, ROUGE) of generative models compare to those of fine-tuned models like T5 and BART?
• RQ3 What types of errors do generative models commonly make when generating text from graphs, particularly regarding semantic relations and factual consistency?
• RQ4 To what extent can machine-generated text be distinguished from human-written text using a fine-tuned BERT classifier?
• RQ5 How reproducible are the outputs of GPT-3 and ChatGPT when prompted with the same input graph?

Key findings

• GPT-3 and ChatGPT achieve BLEU scores of 10.57 and 11.08 on the AGENDA and WebNLG datasets, respectively, indicating fluent but low-quality text generation.
• Generative models frequently fail to understand semantic relations between entities, such as 'USED-FOR' and 'PART-OF', leading to factual inaccuracies.
• ChatGPT generates redundant and hallucinated content, including subjective claims like 'Abner W. Sibal was a prominent figure who faced great challenges'.
• GPT-3 incorrectly identifies names as events, e.g., labeling 'Abner W. Sibal' as a battle, demonstrating factual hallucinations.
• Fine-tuned BERT models achieve macro-F1 scores of 95.82 on the combined dataset, indicating high detectability of machine-generated text.
• The results are limited by low reproducibility, as GPT-3 and ChatGPT often produce different outputs for the same input, affecting evaluation consistency.

(b) Generate text from graph: <H> Auburn Washington <R> is Part Of <T> Pierce County Washington <H> Pierce County Washington <R> country <T> United States