[Paper Review] Towards Theme Detection in Personal Finance Questions

John Xi Qiu, Adam Faulkner+1 more|arXiv (Cornell University)|Oct 4, 2021

Topic Modeling14 references4 citations

TL;DR

This paper proposes a sentence-level clustering approach for theme detection in personal finance queries, using Universal Sentence Encoder (USE) and KMeans clustering on StackExchange data to identify multiple themes per question. The method achieves a Micro-F1 of 0.46, outperforming SBERT and more complex semantic parsing techniques, demonstrating that simple, robust encoding and clustering can effectively surface topical trends in financial customer inquiries.

ABSTRACT

Banking call centers receive millions of calls annually, with much of the information in these calls unavailable to analysts interested in tracking new and emerging call center trends. In this study we present an approach to call center theme detection that captures the occurrence of multiple themes in a question, using a publicly available corpus of StackExchange personal finance questions, labeled by users with topic tags, as a testbed. To capture the occurrence of multiple themes in a single question, the approach encodes and clusters at the sentence- rather than question-level. We also present a comparison of state-of-the-art sentence encoding models, including the SBERT family of sentence encoders. We frame our evaluation as a multiclass classification task and show that a simple combination of the original sentence text, Universal Sentence Encoder, and KMeans outperforms more sophisticated techniques that involve semantic parsing, SBERT-family models, and HDBSCAN. Our highest performing approach achieves a Micro-F1 of 0.46 for this task and we show that the resulting clusters, even when slightly noisy, contain sentences that are topically consistent with the label associated with the cluster.

Motivation & Objective

• To detect multiple emerging themes in personal finance customer inquiries from call center transcripts.
• To address the limitation of keyword-based search by identifying 'unknown unknowns'—previously undetected trends in customer queries.
• To evaluate and compare sentence encoding and clustering techniques for theme detection in a finance-domain context.
• To develop a scalable, unsupervised pipeline that operates at the sentence level to capture multiple themes per question.
• To validate the approach using publicly available personal finance questions from StackExchange as a proxy for proprietary call center data.

Proposed method

• The method processes customer problem statements (CPS) from call transcripts by segmenting them into sentences for analysis.
• It uses sentence-level encoding with Universal Sentence Encoder (USE) and SBERT-family models to generate dense vector representations.
• Clustering is performed using KMeans and HDBSCAN on the encoded sentence vectors to group semantically similar sentences.
• The approach combines raw sentence text, USE embeddings, and KMeans clustering, avoiding complex semantic parsing.
• Clusters are labeled by generalizing lexical patterns in the clustered sentences into human-readable theme labels.
• Evaluation is framed as a multiclass classification task using Micro-F1, with error analysis focusing on misclassification patterns and dominant class bias.

Experimental results

Research questions

• RQ1 Can sentence-level clustering effectively detect multiple themes within a single personal finance question?
• RQ2 How do different sentence encoding models—particularly USE versus SBERT—perform in theme detection for financial queries?
• RQ3 Does using raw sentence text with USE and KMeans outperform more complex approaches involving semantic parsing or advanced models?
• RQ4 What causes misclassification in theme detection, and how does class imbalance affect performance?
• RQ5 Can the proposed method detect emerging trends over time in financial query data?

Key findings

• The combination of Universal Sentence Encoder (USE) and KMeans clustering achieved the highest Micro-F1 score of 0.46, outperforming all SBERT-family and TF-IDF-based baselines.
• SBERT-family models underperformed compared to USE, despite being state-of-the-art, suggesting domain-specific pretraining may be more effective for financial text.
• Sentence-level clustering successfully captured multiple themes per question, with clusters showing strong topical consistency even when slightly noisy.
• HDBSCAN consistently chose fewer clusters (around 100) than KMeans (around 700), limiting its ability to capture topic diversity.
• Misclassifications were often topically related, such as 'trading' being confused with 'investing', indicating semantic similarity in financial domains.
• The dominant 'investing' class caused a high number of false positives, suggesting that future work should refine this class or normalize prediction scores.