[Paper Review] TwistBytes -- Hierarchical Classification at GermEval 2019: walking the fine line (of recall and precision)

Fernando Benites|arXiv (Cornell University)|Aug 18, 2019

Text and Document Classification Technologies5 references4 citations

TL;DR

This paper presents TwistBytes, a traditional NLP approach using TF-IDF with linear SVM for hierarchical multi-label classification in German book blurbs, achieving first place in GermEval 2019's subtask B (hierarchical classification) and second in subtask A (flat classification). The key innovation lies in a post-processing strategy that boosts recall without sacrificing precision, enabling strong performance on a low-cardinality, multi-label hierarchy with limited training data.

ABSTRACT

We present here our approach to the GermEval 2019 Task 1 - Shared Task on hierarchical classification of German blurbs. We achieved first place in the hierarchical subtask B and second place on the root node, flat classification subtask A. In subtask A, we applied a simple multi-feature TF-IDF extraction method using different n-gram range and stopword removal, on each feature extraction module. The classifier on top was a standard linear SVM. For the hierarchical classification, we used a local approach, which was more light-weighted but was similar to the one used in subtask A. The key point of our approach was the application of a post-processing to cope with the multi-label aspect of the task, increasing the recall but not surpassing the precision measure score.

Motivation & Objective

• To address hierarchical multi-label classification of German book blurbs with a scalable, lightweight NLP pipeline.
• To improve recall in multi-label classification without degrading precision, especially in low-cardinality label settings.
• To evaluate the effectiveness of post-processing techniques in hierarchical classification, particularly for imbalanced or sparse label distributions.
• To demonstrate that traditional machine learning methods can outperform deep learning on small-to-medium datasets with complex hierarchies.
• To enhance an open-source library for hierarchical multi-label classification with multi-label support and improved inference.

Proposed method

• Feature extraction using multi-feature TF-IDF with varying n-gram ranges and stopword removal.
• Classification via linear SVM on both flat and hierarchical subtasks, using shared hyperparameters across tasks.
• Application of a threshold-based post-processing step to convert raw prediction scores into final label sets.
• Use of label cardinality from the training set to guide threshold selection, ensuring balanced prediction distribution.
• Adoption of a local classification approach that predicts labels per node rather than global DAG traversal.
• Optimization of the threshold via empirical tuning, with -0.25 yielding the best micro-F1 in subtask B.

Experimental results

Research questions

• RQ1 Can a lightweight, traditional NLP pipeline outperform deep learning models on a small-to-medium German text classification dataset with hierarchical labels?
• RQ2 How effective is post-processing in improving recall without significantly reducing precision in hierarchical multi-label classification?
• RQ3 What thresholding strategy best balances precision and recall in a low-cardinality, multi-label setting?
• RQ4 To what extent can existing feature extraction and classification tools be adapted for hierarchical multi-label tasks with minimal modification?
• RQ5 How does label cardinality influence the choice of classification and post-processing strategies in hierarchical text classification?

Key findings

• TwistBytes achieved the highest micro-F1 score of 0.6767 in subtask B, securing first place in the hierarchical classification task.
• The method achieved the highest recall (0.6487) among all teams in subtask B, indicating effective label coverage despite low label cardinality.
• Precision was maintained at 0.7072, showing a strong balance between precision and recall, with the harmonic mean providing a decisive advantage.
• In subtask A, the approach ranked second with a micro-F1 of 0.8634, demonstrating strong generalization to flat classification.
• The post-processing step significantly improved recall without causing a sharp drop in precision, validating its role in optimizing the F1 score.
• The results suggest that traditional methods like TF-IDF with SVM remain competitive on datasets with limited training samples and complex hierarchies.