[Paper Review] Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus

Daniela Trotta, Raffaele Guarasci+2 more|arXiv (Cornell University)|Sep 24, 2021

Natural Language Processing Techniques53 references27 citations

TL;DR

This paper introduces ItaCoLA, a new Italian corpus of 10,000 sentences with binary acceptability judgments and additional annotations for nine linguistic phenomena, created using the same methodology as the English CoLA. It demonstrates that monolingual BERT models outperform multilingual XLM-RoBERTa in acceptability classification, while cross-lingual fine-tuning shows potential despite lower performance.

ABSTRACT

The development of automated approaches to linguistic acceptability has been greatly fostered by the availability of the English CoLA corpus, which has also been included in the widely used GLUE benchmark. However, this kind of research for languages other than English, as well as the analysis of cross-lingual approaches, has been hindered by the lack of resources with a comparable size in other languages. We have therefore developed the ItaCoLA corpus, containing almost 10,000 sentences with acceptability judgments, which has been created following the same approach and the same steps as the English one. In this paper we describe the corpus creation, we detail its content, and we present the first experiments on this new resource. We compare in-domain and out-of-domain classification, and perform a specific evaluation of nine linguistic phenomena. We also present the first cross-lingual experiments, aimed at assessing whether multilingual transformerbased approaches can benefit from using sentences in two languages during fine-tuning.

Motivation & Objective

• To address the lack of large-scale, high-quality acceptability judgment resources for Italian and other non-English languages.
• To create a standardized, methodologically consistent corpus for Italian linguistic acceptability, mirroring the English CoLA.
• To evaluate monolingual and cross-lingual classification performance using BERT-based and multilingual XLM-RoBERTa models.
• To enable fine-grained analysis of specific linguistic phenomena in acceptability judgments.

Proposed method

• The ItaCoLA corpus was constructed by extracting 10,000 sentences from existing linguistic literature, with acceptability judgments assigned by expert linguists.
• A subset of 30% of sentences was manually annotated with nine linguistic phenomena, including wh-islands, auxiliary violations, and copula constructions.
• Monolingual classification was performed using a BERT-based classifier fine-tuned on the ItaCoLA training split.
• Cross-lingual experiments used XLM-RoBERTa-base, fine-tuned jointly on English CoLA and Italian ItaCoLA training data.
• Model performance was evaluated using Matthews Correlation Coefficient (MCC), with in-domain and out-of-domain test sets.
• Experiments compared monolingual and cross-lingual setups, including zero-shot cross-lingual inference.

Experimental results

Research questions

• RQ1 Can a large-scale, expert-annotated acceptability corpus for Italian be created using the same methodology as the English CoLA?
• RQ2 How does monolingual BERT-based classification performance on ItaCoLA compare to results on the English CoLA?
• RQ3 To what extent can multilingual models like XLM-RoBERTa benefit from joint fine-tuning on English and Italian acceptability data?
• RQ4 How does sentence complexity—particularly multiple linguistic phenomena in one sentence—affect classification performance?
• RQ5 Can cross-lingual transfer learning from English to Italian achieve competitive results on acceptability classification?

Key findings

• The ItaCoLA corpus achieves a Matthews Correlation Coefficient (MCC) of 0.517 on the in-domain test set, comparable to the 0.582 MCC reported for BERT on the English CoLA.
• Monolingual BERT models on ItaCoLA outperform multilingual XLM-RoBERTa, which achieved only 0.440 MCC on the Italian test set.
• Cross-lingual zero-shot inference using XLM-RoBERTa performed poorly, with MCC of 0.114 on both English and Italian test sets.
• Performance drops significantly on sentences with multiple annotated phenomena: MCC fell from 0.363 ± 0.021 (single phenomenon) to 0.308 ± 0.041 (multiple phenomena).
• The best results for specific phenomena were achieved for Wh-islands and Auxiliary violations, while Copula constructions and Simple sentences showed lower performance.
• Fine-tuning on only the Italian subset yielded a 0.440 MCC, outperforming zero-shot cross-lingual inference but still underperforming compared to monolingual BERT on English.