[Paper Review] ALBERTI, a Multilingual Domain Specific Language Model for Poetry Analysis

Javier de la Rosa, Álvaro Pérez Pozo+2 more|arXiv (Cornell University)|Jul 3, 2023

Topic Modeling4 citations

TL;DR

This paper introduces Alberti, the first multilingual domain-specific language model pre-trained on over 12 million poetic verses across 12 languages using domain-specific pre-training (DSP) on multilingual BERT. It achieves state-of-the-art performance on Spanish stanza classification and metrical pattern prediction, outperforming mBERT and even rule-based systems in German, demonstrating the effectiveness of DSP for poetry analysis in multilingual settings.

ABSTRACT

The computational analysis of poetry is limited by the scarcity of tools to automatically analyze and scan poems. In a multilingual settings, the problem is exacerbated as scansion and rhyme systems only exist for individual languages, making comparative studies very challenging and time consuming. In this work, we present extsc{Alberti}, the first multilingual pre-trained large language model for poetry. Through domain-specific pre-training (DSP), we further trained multilingual BERT on a corpus of over 12 million verses from 12 languages. We evaluated its performance on two structural poetry tasks: Spanish stanza type classification, and metrical pattern prediction for Spanish, English and German. In both cases, extsc{Alberti} outperforms multilingual BERT and other transformers-based models of similar sizes, and even achieves state-of-the-art results for German when compared to rule-based systems, demonstrating the feasibility and effectiveness of DSP in the poetry domain.

Motivation & Objective

• To address the scarcity of multilingual computational tools for poetry analysis, especially for scansion and structural analysis.
• To overcome the limitations of monolingual models and rule-based systems that lack cross-lingual generalization and scalability.
• To investigate whether domain-specific pre-training (DSP) on poetic corpora improves performance on poetry-specific NLP tasks in multiple languages.
• To release a multilingual poetry-focused language model and dataset to support future research in digital humanities and NLP.

Proposed method

• Fine-tuned multilingual BERT (mBERT) on a curated corpus of over 12 million poetic verses spanning 12 languages using masked language modeling (MLM) for domain-specific pre-training.
• Reformulated poetry analysis tasks—such as stanza type classification and metrical pattern prediction—as text classification problems to leverage transfer learning.
• Trained the model on a multilingual poetry dataset derived from public sources, including poetic forms from Spanish, English, German, and other languages.
• Evaluated intrinsic performance via perplexity on a multilingual poetry benchmark (PULPO), and extrinsic performance on downstream classification tasks.
• Used ROC-AUC and F1 scores to compare Alberti’s performance against mBERT and other baselines on stanza classification and scansion prediction.
• Released both the Alberti model weights and the training corpus to enable reproducibility and further research in multilingual poetry NLP.

Experimental results

Research questions

• RQ1 Can domain-specific pre-training on multilingual poetic corpora improve performance on poetry-specific NLP tasks compared to general multilingual models?
• RQ2 Does a multilingual language model fine-tuned on poetry outperform mBERT and other transformer-based models in stanza type classification and metrical pattern prediction?
• RQ3 To what extent can a single multilingual model generalize across diverse poetic forms and languages, including those with complex scansion rules like German and Spanish?
• RQ4 Can a domain-specific language model trained on poetic text achieve state-of-the-art results in metrical pattern prediction, even when compared to rule-based systems?
• RQ5 How effective is the model in capturing linguistic and structural nuances of poetry across languages, as measured by intrinsic and extrinsic evaluation metrics?

Key findings

• Alberti outperformed mBERT on the PULPO perplexity benchmark, achieving a lower average perplexity of 83.31 compared to mBERT’s 128.64, indicating better language modeling on poetic text.
• On Spanish stanza type classification, Alberti achieved an average AUC score of 0.9857 for Tercetillo and up to 0.9998 for Rom. arte mayor, significantly outperforming mBERT’s best AUC of 0.9857.
• For metrical pattern prediction, Alberti achieved state-of-the-art results in German, surpassing rule-based systems, with an AUC of 0.9981 for the Lira form.
• Alberti achieved an F1-score of 0.9767 on the Cuarteto form and 0.9975 on the Copla mixta form, demonstrating strong generalization across diverse poetic structures.
• The model showed strong cross-lingual transferability, with consistent improvements across all 12 languages in both intrinsic (perplexity) and extrinsic (classification) evaluations.
• The release of Alberti and its training corpus provides a foundational resource for future multilingual poetry analysis, enabling large-scale comparative studies across linguistic traditions.