[Paper Review] The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild

Taja Kuzman, Peter Rupnik+1 more|arXiv (Cornell University)|Jan 11, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces GINCO 1.0, a manually annotated Slovenian web corpus of 1,125 documents (650k words) for automatic genre identification, featuring a new schema emphasizing label clarity and high inter-annotator agreement. Experiments show Transformer-based models (e.g., SloBERTa, XLM-RoBERTa) achieve macro F1 scores of ~0.58, significantly outperforming non-Transformer models (~0.22), while reducing label classes from 21 to 12 boosts macro F1 to 0.668.

ABSTRACT

This paper presents a new training dataset for automatic genre identification GINCO, which is based on 1,125 crawled Slovenian web documents that consist of 650 thousand words. Each document was manually annotated for genre with a new annotation schema that builds upon existing schemata, having primarily clarity of labels and inter-annotator agreement in mind. The dataset consists of various challenges related to web-based data, such as machine translated content, encoding errors, multiple contents presented in one document etc., enabling evaluation of classifiers in realistic conditions. The initial machine learning experiments on the dataset show that (1) pre-Transformer models are drastically less able to model the phenomena, with macro F1 metrics ranging around 0.22, while Transformer-based models achieve scores of around 0.58, and (2) multilingual Transformer models work as well on the task as the monolingual models that were previously proven to be superior to multilingual models on standard NLP tasks.

Motivation & Objective

• To develop a realistic, high-quality training dataset for automatic genre identification in web documents, particularly for low-resource languages like Slovenian.
• To address challenges in web-based data such as machine translation, encoding errors, and hybrid genres through a curated, manually annotated corpus.
• To improve genre annotation reliability by designing a schema focused on label clarity and high inter-annotator agreement (Krippendorff’s alpha = 0.71).
• To evaluate the performance of various neural architectures on genre classification in realistic, noisy web conditions.
• To enable future cross-lingual genre classification by extending the dataset to Croatian and English.

Proposed method

• Annotate 1,125 Slovenian web documents with a new genre classification schema that integrates purpose, form, and lexico-grammatical features to improve label clarity and inter-annotator agreement.
• Construct the GINCO 1.0 dataset by sampling from two existing Slovenian web corpora, preserving real-world noise such as machine-translated content, encoding errors, and multi-genre documents.
• Apply a two-step data filtering: remove near-duplicate paragraphs and retain only the dominant genre label per document to improve model generalization.
• Train and evaluate multiple models, including CNN-based, monolingual (SloBERTa), and multilingual (XLM-RoBERTa) Transformer models, on the final dataset.
• Perform ablation studies on label set size by merging 21 labels into 12, focusing on improving performance for low-resource classes.
• Use statistical testing (Mann-Whitney U test) to validate significance of performance differences across configurations.

Experimental results

Research questions

• RQ1 Can a new genre classification schema achieve high inter-annotator agreement while maintaining label clarity for web documents?
• RQ2 How do different neural architectures perform on genre classification in realistic, noisy web corpora?
• RQ3 Does reducing the number of genre classes improve classification performance, especially for low-resource categories?
• RQ4 To what extent do near-duplicate paragraphs and multi-genre documents negatively impact model performance?
• RQ5 Can multilingual models perform as well as monolingual models on a low-resource language like Slovenian in genre identification tasks?

Key findings

• Pre-Transformer models achieve a macro F1 score of approximately 0.22, indicating poor performance on the GINCO dataset due to limited modeling capacity for complex web genres.
• Transformer-based models, including both monolingual SloBERTa and multilingual XLM-RoBERTa, achieve a macro F1 score of around 0.58, demonstrating the necessity of contextualized representations for this task.
• Reducing the number of genre labels from 21 to 12 significantly improves macro F1 from 0.575 to 0.668 (p < 0.001), especially benefiting underrepresented classes like 'Legal/Regulation' and 'Other'.
• The confusion matrices show clearer class separation and reduced misclassification after label reduction, particularly for 'Promotion' and 'Other' categories.
• Micro F1 improves from 0.629 (21 labels) to 0.696 (12 labels), with statistical significance confirmed via Mann-Whitney U test (p < 0.001).
• The study identifies 10.9% of the original data as unsuitable for genre classification due to ambiguity or complexity, highlighting the need for better preprocessing or automated splitting of multi-genre documents.