[Paper Review] Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs

Alejandro Peña, Aythami Morales+6 more|arXiv (Cornell University)|Jun 5, 2023

Computational and Text Analysis Methods19 references4 citations

TL;DR

This paper proposes a multi-label topic classification system for Spanish public affairs documents using fine-tuned Large Language Models (LLMs) and hybrid classifiers. It demonstrates that RoBERTa-based LLMs combined with SVM classifiers achieve superior performance—especially in low-data regimes—outperforming neural networks and random forests, with F1 scores exceeding 80% on most topics despite class imbalance.

ABSTRACT

The analysis of public affairs documents is crucial for citizens as it promotes transparency, accountability, and informed decision-making. It allows citizens to understand government policies, participate in public discourse, and hold representatives accountable. This is crucial, and sometimes a matter of life or death, for companies whose operation depend on certain regulations. Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents by effectively processing and understanding the complex language used in such documents. In this work, we analyze the performance of LLMs in classifying public affairs documents. As a natural multi-label task, the classification of these documents presents important challenges. In this work, we use a regex-powered tool to collect a database of public affairs documents with more than 33K samples and 22.5M tokens. Our experiments assess the performance of 4 different Spanish LLMs to classify up to 30 different topics in the data in different configurations. The results shows that LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.

Motivation & Objective

• To improve automatic analysis of public affairs documents for citizens and businesses affected by regulatory changes.
• To address the challenge of multi-label topic classification in Spanish legislative texts with limited labeled data.
• To evaluate the effectiveness of recent Spanish LLMs and hybrid models (LLM + classifier) in classifying public affairs documents.
• To develop a scalable pipeline for monitoring regulatory changes using NLP and document layout analysis.
• To identify optimal LLM and classifier combinations for domain-specific topic classification in public administration.

Proposed method

• Collected a Spanish corpus of 33,000+ multi-labeled public affairs documents from Spanish parliamentary initiatives over two years using a regex-powered annotation tool.
• Preprocessed and curated the dataset, selecting the 30 most frequent topics from 385 total topics annotated by domain experts.
• Fine-tuned three RoBERTa-based encoder models and one generative LLM for text understanding and embedding generation.
• Combined LLM embeddings with three classifiers: Neural Networks, Random Forest, and SVM for multi-label topic classification.
• Applied document layout analysis (DLA) using Random Forest to extract semantic text blocks before LLM processing.
• Used a three-module system: web harvester, DLA module for block classification, and LLM-based text processing for topic inference.

Experimental results

Research questions

• RQ1 How effective are Spanish LLMs in classifying multi-label topics in public affairs documents with limited training data?
• RQ2 Which combination of LLM and downstream classifier yields the best performance for topic classification in this domain?
• RQ3 How do class imbalance and low sample size affect the performance of different LLM-based classification setups?
• RQ4 Can SVMs mitigate bias toward the negative class in low-resource topic classification compared to neural networks and random forests?
• RQ5 To what extent do RoBERTa-based models outperform generative LLMs in this zero-shot or few-shot topic classification task?

Key findings

• SVM classifiers combined with RoBERTa-base embeddings achieved the highest TPR (True Positive Rate) across all topics, with mean TPR exceeding 80% on most topics, even with fewer than 2,000 samples.
• Random Forest classifiers performed poorly, especially with low-resource topics, often predicting only the negative class due to class imbalance and poor generalization on embedding spaces.
• Neural network classifiers performed comparably to SVMs on high-resource topics but showed higher variance and lower robustness in low-data settings.
• The SVM-RoBERTa combination reduced bias toward the negative class compared to Random Forest and neural networks, particularly in topics with fewer than 2,000 samples.
• For topic 15, the SVM model achieved a 70% TPR, significantly outperforming the RoBERTa-NN model (61% TPR), indicating superior handling of imbalanced data.
• The results suggest that SVMs are more effective than neural networks and Random Forests for fine-tuning LLM embeddings in low-data, multi-label classification tasks in public affairs.