[Paper Review] From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French

Simon Gabay, Pedro Ortiz Suárez+5 more|arXiv (Cornell University)|Feb 18, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces FreEM max, a large-scale corpus of Early Modern French (16th–18th centuries), and D’AlemBERT, a RoBERTa-based language model pre-trained on this corpus. Fine-tuning on part-of-speech tagging tasks shows state-of-the-art performance, with strong transfer learning effects from more-resourced time periods to under-resourced ones, demonstrating the model's effectiveness for low-resource historical NLP tasks.

ABSTRACT

Language models for historical states of language are becoming increasingly important to allow the optimal digitisation and analysis of old textual sources. Because these historical states are at the same time more complex to process and more scarce in the corpora available, specific efforts are necessary to train natural language processing (NLP) tools adapted to the data. In this paper, we present our efforts to develop NLP tools for Early Modern French (historical French from the 16$^ ext{th}$ to the 18$^ ext{th}$ centuries). We present the $ ext{FreEM}_{ ext{max}}$ corpus of Early Modern French and D'AlemBERT, a RoBERTa-based language model trained on $ ext{FreEM}_{ ext{max}}$. We evaluate the usefulness of D'AlemBERT by fine-tuning it on a part-of-speech tagging task, outperforming previous work on the test set. Importantly, we find evidence for the transfer learning capacity of the language model, since its performance on lesser-resourced time periods appears to have been boosted by the more resourced ones. We release D'AlemBERT and the open-sourced subpart of the $ ext{FreEM}_{ ext{max}}$ corpus.

Motivation & Objective

• Address the scarcity and complexity of linguistic resources for Early Modern French, a historically significant but computationally challenging language state.
• Overcome the paradox that historical languages with greater linguistic variation are less resourced and harder to process than modern ones.
• Develop a high-quality, large-scale corpus (FreEM max) and a contextualized language model (D’AlemBERT) tailored for Early Modern French.
• Demonstrate the utility of transfer learning in historical NLP by evaluating the model on low-resource time periods within the same language state.
• Release both the D’AlemBERT model and a subpart of the FreEM max corpus to support open research in digital humanities and historical NLP.

Proposed method

• Constructed FreEM max, a large, curated corpus of Early Modern French texts spanning the 16th to 18th centuries, sourced from multiple digitized archives and historical collections.
• Pre-trained a RoBERTa-based language model, D’AlemBERT, on the FreEM max corpus using masked language modeling objective.
• Applied transfer learning by fine-tuning D’AlemBERT on a part-of-speech tagging task for Early Modern French, using a standard benchmark dataset.
• Evaluated model performance across different time periods, including both well-resourced and under-resourced sub-periods, to assess transferability.
• Implemented standard NLP evaluation metrics (e.g., accuracy, F1-score) to compare D’AlemBERT against prior state-of-the-art models.
• Released the D’AlemBERT model weights and a subpart of the FreEM max corpus under open-source licenses to enable reproducibility and further research.

Experimental results

Research questions

• RQ1 Can a large, curated corpus of Early Modern French support the training of a high-performing contextual language model?
• RQ2 Does a RoBERTa-based model pre-trained on FreEM max outperform previous models on downstream NLP tasks like part-of-speech tagging?
• RQ3 To what extent does transfer learning from more-resourced time periods improve performance on under-resourced sub-periods within Early Modern French?
• RQ4 Can the proposed model and corpus serve as a foundation for future NLP tasks in historical French, such as named entity recognition or coreference resolution?
• RQ5 What is the impact of training on a historically variable, non-standardized language on model generalization and robustness?

Key findings

• D’AlemBERT outperformed previous state-of-the-art models on the part-of-speech tagging benchmark for Early Modern French, achieving higher accuracy and F1-score.
• The model demonstrated strong transfer learning capacity, with performance on under-resourced time periods significantly improved when trained on more-resourced sub-periods.
• The FreEM max corpus, though not fully open-sourced, includes a substantial subpart released openly, enabling broader access to high-quality historical French text data.
• The RoBERTa-based architecture proved effective for handling the morphosyntactic and orthographic variation typical of Early Modern French.
• The results validate the feasibility and value of training specialized language models for historical languages with limited, heterogeneous resources.
• The release of D’AlemBERT and a portion of FreEM max provides a new benchmark and foundation for future research in digital humanities and historical NLP.