[Paper Review] Texygen: A Benchmarking Platform for Text Generation Models

Yaoming Zhu, Sidi Lu+5 more|arXiv (Cornell University)|Feb 6, 2018

Topic ModelingComputer Science16 references155 citations

TL;DR

Texygen is an open-source benchmarking platform that provides baseline text-generation models and multi-faceted metrics to evaluate diversity, quality, and consistency, aiming to standardize and improve reproducibility in open-domain text generation research.

ABSTRACT

We introduce Texygen, a benchmarking platform to support research on open-domain text generation models. Texygen has not only implemented a majority of text generation models, but also covered a set of metrics that evaluate the diversity, the quality and the consistency of the generated texts. The Texygen platform could help standardize the research on text generation and facilitate the sharing of fine-tuned open-source implementations among researchers for their work. As a consequence, this would help in improving the reproductivity and reliability of future research work in text generation.

Motivation & Objective

• Address the lack of a standardized, multi-faceted evaluation for open-domain text generation.
• Provide an open-source platform with baseline models and reproducible evaluation protocols.
• Encourage comprehensive benchmarking across diversity, quality, and consistency of generated text.

Proposed method

• Implements likelihood-based and adversarial text-generation baselines (vanilla MLE, SeqGAN, MaliGAN, RankGAN, TextGAN, GSGAN, LeakGAN).
• Defines and computes metrics including BLEU, EmbSim, NLLoracle, NLLtest, and Self-BLEU to evaluate quality, similarity to data, and diversity.
• Presents a TensorFlow-based architecture with a GAN class and Oracle interfaces, enabling synthetic and real-data training regimes.
• Uses pretraining via MLE before adversarial training; describes training schedules for different baselines (including LeakGAN’s periodic MLE fine-tuning).
• Offers an open-source repository with API specifications to facilitate model integration and benchmarking.

Experimental results

Research questions

• RQ1 How can open-domain text generation models be evaluated in a standardized, multi-metric framework?
• RQ2 Can a unified platform facilitate reproducibility and sharing of open-source implementations for text generation?
• RQ3 How do baseline models compare in terms of quality, diversity, and consistency on synthetic versus real data?
• RQ4 What metrics best capture diversity and mode collapse in open-domain text generation?
• RQ5 What insights emerge from evaluating both likelihood-based and adversarial approaches within a common platform?

Key findings

• LeakGAN converges quickly and achieves strong performance on NLLoracle and NLLtest in synthetic data experiments.
• On real data, LeakGAN achieves high EmbSim early, while TextGAN improves over pretraining but shows slower EmbSim gains; GSGAN failed to generate semantically meaningful sentences in the real-data setup.
• BLEU results on test data show LeakGAN leading in several BLEU scores (BLEU-2 to BLEU-5) among the baselines, with MLE and other models trailing.
• Self-BLEU indicates mode collapse tendencies for LeakGAN and TextGAN, while MLE and MaliGAN exhibit higher diversity relative to others.
• GSGAN underperformed in semantic quality in the real-data setup and was excluded from some analyses due to lack of meaningful output.
• The platform enables systematic comparison across models and metrics, highlighting strengths and weaknesses of adversarial vs. likelihood-based approaches.