[Paper Review] Czech Text Document Corpus v 2.0

Pavel Král, Ladislav Lenc|arXiv (Cornell University)|Oct 6, 2017

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces Czech Text Document Corpus v 2.0, a freely available, morphologically annotated collection of 11,955 real news articles from the Czech News Agency, designed for benchmarking multi-label document classification in Czech. The corpus supports evaluation using standard metrics and reports state-of-the-art results for methods including maximum entropy, neural networks (MLP and CNN), and hybrid feature approaches, with CNN achieving an F-measure of 84.7% on multi-label classification.

ABSTRACT

This paper introduces "Czech Text Document Corpus v 2.0", a collection of text documents for automatic document classification in Czech language. It is composed of the text documents provided by the Czech News Agency and is freely available for research purposes at http://ctdc.kiv.zcu.cz/. This corpus was created in order to facilitate a straightforward comparison of the document classification approaches on Czech data. It is particularly dedicated to evaluation of multi-label document classification approaches, because one document is usually labelled with more than one label. Besides the information about the document classes, the corpus is also annotated at the morphological layer. This paper further shows the results of selected state-of-the-art methods on this corpus to offer the possibility of an easy comparison with these approaches.

Motivation & Objective

• To provide a standardized, publicly available corpus for evaluating document classification methods in the Czech language.
• To support the development and benchmarking of multi-label classification approaches, as documents are typically assigned multiple labels.
• To facilitate comparison across methods by including morphological annotations (lemmatization, POS, parsing) and pre-trained baselines.
• To enable reproducible evaluation through a defined five-fold cross-validation protocol and hyperparameter tuning using a development set.
• To support future integration into the LINDAT/CLARIN language resource infrastructure.

Proposed method

• The corpus is constructed from 11,955 real newspaper articles provided by the Czech News Agency (CTK), with an additional 2,735 articles for development set tuning.
• Documents are annotated with up to eight labels per article, covering 604 categories, with only the 37 most frequent categories used for classification.
• Morphological annotation is performed automatically using the UDPipe tool, including lemmatization, POS tagging, and syntactic parsing, stored in .conll, .lemma, .pos, and .tok formats.
• Multi-label classification is evaluated using precision, recall, and F-measure (Fm), while single-label classification uses accuracy.
• Experiments use five-fold cross-validation with 20% of the corpus reserved for testing and the development set for hyperparameter tuning.
• State-of-the-art methods evaluated include maximum entropy (ME), hybrid word and unsupervised features (HAL, COALS), and deep learning models (MLP and CNN) with thresholding for multi-label output.

Experimental results

Research questions

• RQ1 What is the performance of standard multi-label classification methods on a large-scale, real-world Czech news corpus with multi-annotated documents?
• RQ2 How do morphological annotations (lemmatization, POS, parsing) impact the performance of document classification models in Czech?
• RQ3 Can neural network-based models (MLP and CNN) outperform traditional machine learning approaches (e.g., ME, SVM) on this corpus for multi-label classification?
• RQ4 How effective are unsupervised features (e.g., LDA, HAL, COALS) when combined with word-level features in improving classification accuracy?
• RQ5 What is the optimal baseline performance for single-label classification on this corpus, and how does it compare to multi-label setups?

Key findings

• The CNN-based model achieved the highest F-measure of 84.7% on the multi-label classification task, outperforming other methods.
• The hybrid feature approach combining unsupervised features (HAL, COALS) with word features achieved an F-measure of 81.7%, demonstrating the value of semantic and distributional features.
• The maximum entropy classifier with word features achieved an F-measure of 76.8%, serving as a strong baseline for traditional methods.
• The MLP model achieved an F-measure of 83.9%, showing that feedforward networks can effectively model multi-label classification in Czech.
• For single-label classification, the SVM classifier achieved 91.2% accuracy, indicating strong performance when only the primary label is considered.
• The corpus exhibits a high degree of label sparsity, with an average of 2.55 labels per document and 80% of documents having two labels.