[Paper Review] Ashaar: Automatic Analysis and Generation of Arabic Poetry Using Deep Learning Approaches

Zaid Alyafeai, Maged S. Al-Shaibani+1 more|arXiv (Cornell University)|Jul 12, 2023

Topic Modeling4 citations

TL;DR

This paper introduces Ashaar, a deep learning framework for automatic analysis and conditional generation of Arabic poetry, leveraging four public datasets and five pre-trained models for tasks including meter classification, diacritization, Arudi style prediction, and theme-conditioned poetry generation. The system achieves strong performance, with a 71.13% top-5 accuracy in meter classification on generated poetry and a 50.28% diacritization error rate in zero-shot evaluation, demonstrating robust generative and analytical capabilities for Arabic poetic structures.

ABSTRACT

Poetry holds immense significance within the cultural and traditional fabric of any nation. It serves as a vehicle for poets to articulate their emotions, preserve customs, and convey the essence of their culture. Arabic poetry is no exception, having played a cherished role in the heritage of the Arabic community throughout history and maintaining its relevance in the present era. Typically, comprehending Arabic poetry necessitates the expertise of a linguist who can analyze its content and assess its quality. This paper presents the introduction of a framework called extit{Ashaar} https://github.com/ARBML/Ashaar, which encompasses a collection of datasets and pre-trained models designed specifically for the analysis and generation of Arabic poetry. The pipeline established within our proposed approach encompasses various aspects of poetry, such as meter, theme, and era classification. It also incorporates automatic poetry diacritization, enabling more intricate analyses like automated extraction of the extit{Arudi} style. Additionally, we explore the feasibility of generating conditional poetry through the pre-training of a character-based GPT model. Furthermore, as part of this endeavor, we provide four datasets: one for poetry generation, another for diacritization, and two for Arudi-style prediction. These datasets aim to facilitate research and development in the field of Arabic poetry by enabling researchers and enthusiasts to delve into the nuances of this rich literary tradition.

Motivation & Objective

• To address the lack of standardized tools and datasets for analyzing and generating Arabic poetry using deep learning.
• To enable automated analysis of poetic elements such as meter, theme, era, and Arudi structure through specialized models.
• To facilitate conditional poetry generation by conditioning on meter, rhyme (qafiyah), and theme using a fine-tuned character-based GPT model.
• To provide open-access, high-quality datasets for meter, diacritization, Arudi representation, and tafeelat to support future research in Arabic computational linguistics.
• To evaluate the generative and analytical performance of models in zero-shot and few-shot settings, particularly in diacritization and rhythm preservation.

Proposed method

• The framework employs a pipeline integrating meter classification, diacritization, and Arudi-style prediction using transformer-based and character-level models.
• A character-based GPT model is fine-tuned for conditional poetry generation, conditioned on specified meter, rhyme (qafiyah), and theme.
• Four public datasets are curated: Ashaar (labeled for meter, theme, era), Ashaar diacritized (cleaned, diacritized poems), Ashaar arudi (gold Arudi representations), and Ashaar tafeelah (complete tafeelat for each meter).
• The diacritization model is evaluated in zero-shot settings using sampling strategies to assess WER and DER metrics without fine-tuning.
• Meter classification for generated poetry is evaluated via majority voting across 1,500 generated poems, with top-n accuracy measured at n=1, 3, and 5.
• The system uses a modified GPT-2 architecture with 10 layers, trained on a character-level tokenizer to preserve morphological and diacritical details in Arabic script.

Figure 1: General pipeline for Ashaar.

Experimental results

Research questions

• RQ1 Can a deep learning framework effectively automate the analysis of Arabic poetic structures, including meter, theme, and era?
• RQ2 To what extent can a pre-trained character-based GPT model generate coherent and rhythmically accurate poetry conditioned on meter, rhyme, and theme?
• RQ3 How well can a zero-shot diacritization model perform on unseen Arabic poetic text without fine-tuning?
• RQ4 What is the performance of the system in predicting Arudi-style structures from poetic verses?
• RQ5 How does the framework compare to state-of-the-art methods in rhythm evaluation and meter classification on generated poetry?

Key findings

• The framework achieves a 71.13% top-5 accuracy in meter classification for generated poetry, outperforming prior work despite a smaller model size.
• The zero-shot diacritization model achieves a 50.28% diacritization error rate (DER) when sampling with a probability of 7, indicating moderate but promising performance without fine-tuning.
• The system demonstrates strong performance in zero-shot diacritization, with WER and DER remaining below 92% and 50% respectively across various sampling probabilities.
• The confusion matrix for meter classification on generated poetry shows that more popular meters are predicted with higher accuracy, with over 50% of meters achieving more than 90% confidence.
• The model achieves a 64.40% top-1 accuracy in rhythm evaluation, surpassing the 56.70% of a prior BPE-tokenized model, indicating better encoding of rhythmic structure.
• The curated datasets—Ashaar, Ashaar diacritized, Ashaar arudi, and Ashaar tafeelah—are publicly released, enabling reproducibility and future research in Arabic poetic NLP.

Figure 3: Confusion matrix for era classification.