[Paper Review] KIND: an Italian Multi-Domain Dataset for Named Entity Recognition

Teresa Paccosi, Alessio Palmero Aprosio|arXiv (Cornell University)|Dec 30, 2021

Topic Modeling4 citations

TL;DR

This paper introduces KIND, a multilingual Italian Named Entity Recognition (NER) dataset with over one million tokens, including 600K manually annotated examples across news, literature, and political discourse. It is the largest gold-standard Italian NER resource to date, supporting training and evaluation of NLP models in low-resource settings, and is freely available under a CC BY-NC 4.0 license via GitHub.

ABSTRACT

In this paper we present KIND, an Italian dataset for Named-entity recognition. It contains more than one million tokens with annotation covering three classes: person, location, and organization. The dataset (around 600K tokens) mostly contains manual gold annotations in three different domains (news, literature, and political discourses) and a semi-automatically annotated part. The multi-domain feature is the main strength of the present work, offering a resource which covers different styles and language uses, as well as the largest Italian NER dataset with manual gold annotations. It represents an important resource for the training of NER systems in Italian. Texts and annotations are freely downloadable from the Github repository.

Motivation & Objective

• To address the scarcity of high-quality, manually annotated Italian NER datasets for training and evaluating NLP models.
• To create a multilingual, multi-domain resource that captures linguistic variation across different text styles and genres in Italian.
• To provide a freely accessible, high-quality dataset to support research and development in Italian NLP, especially in low-resource settings.
• To extend existing NER resources by including diverse domains such as literature and political discourse, not just news.
• To support the development of robust, domain-generalizable NER systems for Italian through a balanced, multi-domain corpus.

Proposed method

• The dataset was constructed using publicly available texts under permissive licenses, including Wikinews articles, Italian fiction books, and speeches from Italian politicians Aldo Moro and Alcide De Gasperi.
• Manual annotation was performed by expert linguists on 600K tokens from news, literature, and political discourse, covering three entity types: person (PER), organization (ORG), and location (LOC).
• A semi-automated pipeline was used to annotate approximately 400K tokens from Aldo Moro’s writings, leveraging existing annotations converted via mapping rules and guidelines.
• The annotation process followed consistent guidelines across domains, ensuring consistency in entity labeling despite stylistic differences.
• Texts were collected from trusted public sources: Wikinews (CC BY 2.5), Liber Liber (public domain), and official archives of Italian political figures.
• The dataset is released under the CC BY-NC 4.0 license (with CC BY-NC-SA 4.0 for De Gasperi’s texts), enabling non-commercial research use and distribution via GitHub.

Experimental results

Research questions

• RQ1 Can a large-scale, multilingual, and multi-domain NER dataset be constructed for Italian with high-quality gold-standard annotations?
• RQ2 How does the performance of NER models trained on KIND compare across diverse domains such as news, literature, and political discourse?
• RQ3 To what extent does the inclusion of non-news domains (e.g., literature and political speeches) improve the robustness and generalization of Italian NER systems?
• RQ4 How effective is a semi-automated annotation pipeline in preserving annotation quality when applied to historical or formal political texts?
• RQ5 Can KIND serve as a benchmark for evaluating and improving multilingual and low-resource NLP systems in Italian?

Key findings

• KIND contains over one million tokens, with 600,000 manually annotated tokens across three domains: news (308,622 tokens), literature (192,448 tokens), and political discourse (543,240 tokens).
• The dataset includes 10,665 person, 14,955 organization, and 15,013 location entities in the manually annotated portion, with balanced distribution across domains.
• The corpus is the largest gold-standard Italian NER dataset to date, surpassing previous benchmarks like I-CAB (180K words) and MEANTIME (480 articles).
• The semi-automated annotation of Aldo Moro’s texts (392,604 tokens) achieved a high level of consistency, enabling the inclusion of formal political discourse with minimal human effort.
• The dataset is freely available under CC BY-NC 4.0, with full text and annotations accessible via the GitHub repository, supporting reproducibility and community use.
• The inclusion of literature and political discourse expands the linguistic diversity of Italian NER resources, enabling models to generalize beyond journalistic language.