[Paper Review] GPT-based Generation for Classical Chinese Poetry

Yi Liao, Yasheng Wang+2 more|arXiv (Cornell University)|Jun 29, 2019

Topic Modeling14 references34 citations

TL;DR

A simple GPT-based approach fines-tunes a transformer on classical Chinese poetry to generate well-formed poems across multiple forms, including acrostics, without hand-crafted rules. The system uses token-by-token sampling and form/theme formatting to produce diverse, coherent poetry.

ABSTRACT

We present a simple yet effective method for generating high quality classical Chinese poetry with Generative Pre-trained Language Model (GPT). The method adopts a simple GPT model, without using any human crafted rules or features, or designing any additional neural components. While the proposed model learns to generate various forms of classical Chinese poems, including Jueju, Lüshi, various Cipai and Couples, the generated poems are of very high quality. We also propose and implement a method to fine-tune the model to generate acrostic poetry. To the best of our knowledge, this is the first to employ GPT in developing a poetry generation system. We have released an online mini demonstration program on Wechat to show the generation capability of the proposed method for classical Chinese poetry.

Motivation & Objective

• Demonstrate that a simple GPT model can generate high-quality classical Chinese poetry without explicit forms or content rules.
• Investigate the model's ability to handle multiple forms (Jueju, Lüshi, Cipai, Couplet) and acrostic poetry.
• Evaluate form well-formedness, coherence, diversity, and artistry of generated poems across forms and themes.

Proposed method

• Fine-tune a GPT model on a corpus of classical Chinese poetry after pre-training on a Chinese news corpus.
• Serialize training poems into sequences with fields form, theme, and body, separated by identifiers.
• Use token-by-token generation with truncated top-k sampling to promote diversity during decoding.
• Train a separate model variant to generate acrostic poetry by altering the sequence format to encode head characters.
• Do not impose hard constraints during decoding; rely on the model to learn form and rhyming tendencies.

Experimental results

Research questions

• RQ1 Can a GPT model fine-tuned on classical Chinese poetry generate well-formed poems across various traditional forms without explicit rule-based controllers?
• RQ2 How well can the model learn pairing (duìliǎng) and other form- and tone-related constraints implicitly?
• RQ3 Is the model capable of generating acrostic poetry while ensuring specified tokens appear in designated positions?
• RQ4 What is the diversity, coherence, and artistic quality of poems produced for different themes and forms?

Key findings

• The method generates high-quality poems across multiple forms including Jueju, Lüshi, Cipai, and Couplet without handcrafted rules.
• Well-formedness exceeds 95% for Jueju and Lüshi in the reported observations, with Cipai performance varying by form complexity and data availability.
• The model implicitly learns pairing rules and tone patterns, yielding well-paired lines in many cases, especially for Lüshi.
• Diversity is achieved through truncated top-k sampling, producing different outputs for the same form and theme across runs.
• Acrostic poetry can be generated by a simple reformatting of the input, guiding specific characters to appear in designated positions.
• Rhyming and tone patterns emerge fairly well despite not being explicitly constrained during decoding.