[Paper Review] Large Language Models Enable Few-Shot Clustering

Vijay Viswanathan, Kiril Gashteovski+3 more|arXiv (Cornell University)|Jul 2, 2023

Topic Modeling4 citations

TL;DR

This paper proposes using large language models (LLMs) to enable few-shot semi-supervised text clustering by leveraging LLMs to generate pseudo-feedback—such as keyphrase expansions, pairwise constraints, or post-corrections—reducing human annotation costs while improving cluster quality. LLMs significantly enhance clustering performance with minimal user input, outperforming traditional K-Means and matching human-annotated results at a fraction of the cost.

ABSTRACT

Unlike traditional unsupervised clustering, semi-supervised clustering allows users to provide meaningful structure to the data, which helps the clustering algorithm to match the user's intent. Existing approaches to semi-supervised clustering require a significant amount of feedback from an expert to improve the clusters. In this paper, we ask whether a large language model can amplify an expert's guidance to enable query-efficient, few-shot semi-supervised text clustering. We show that LLMs are surprisingly effective at improving clustering. We explore three stages where LLMs can be incorporated into clustering: before clustering (improving input features), during clustering (by providing constraints to the clusterer), and after clustering (using LLMs post-correction). We find incorporating LLMs in the first two stages can routinely provide significant improvements in cluster quality, and that LLMs enable a user to make trade-offs between cost and accuracy to produce desired clusters. We release our code and LLM prompts for the public to use.

Motivation & Objective

• To reduce the high annotation cost of traditional semi-supervised clustering by using LLMs to amplify expert feedback.
• To investigate whether LLMs can serve as effective pseudo-oracles for pairwise constraints in clustering.
• To evaluate the impact of LLM-augmented document representations on clustering quality across diverse text clustering tasks.
• To compare the cost-effectiveness of LLM-guided clustering versus human-annotated clustering.
• To explore the minimal prompt requirements (e.g., instruction-only vs. demonstrations) for effective LLM integration in clustering.

Proposed method

• LLMs generate keyphrases from input documents, which are embedded and concatenated to original document embeddings to enrich representations before clustering.
• An LLM acts as a pairwise constraint pseudo-oracle, generating similarity judgments between document pairs to guide PCKMeans clustering.
• Post-clustering, the LLM re-evaluates low-confidence cluster assignments using the same pairwise constraint oracle to correct errors.
• The method is applied to three tasks: entity canonicalization, intent clustering, and topic grouping using five benchmark datasets.
• All LLM interactions are driven by natural language prompts, enabling plug-in integration with any text clustering pipeline.
• Cost and performance are evaluated using GPT-3.5-turbo-0301 API, with metrics including Macro F1, Pairwise F1, and Micro F1.

Figure 1: In traditional semi-supervised clustering, a user provides a large amount of feedback to the clusterer. In our approach, the user prompts an LLM with a small amount of feedback. The LLM then generates a large amount of pseudo-feedback for the clusterer.

Experimental results

Research questions

• RQ1 Can LLMs effectively generate pseudo-feedback to reduce human annotation cost in semi-supervised text clustering?
• RQ2 How does LLM-augmented document representation compare to traditional embedding methods in clustering quality?
• RQ3 Does using an LLM as a pairwise constraint pseudo-oracle outperform human-annotated constraints in cost-performance trade-offs?
• RQ4 What is the minimal prompt input (e.g., instruction-only vs. demonstrations) required for effective LLM guidance in clustering?
• RQ5 How effective is LLM-based post-correction in improving cluster assignments after initial clustering?

Key findings

• LLM keyphrase expansion improved clustering quality on all five datasets across all metrics, with gains of up to 5.2 absolute points in F1.
• Using an LLM as a pairwise constraint pseudo-oracle achieved performance comparable to a true human oracle, especially when sufficient feedback (e.g., 20K pairs) was generated.
• LLM post-correction provided only marginal gains, reassigning fewer than 10% of the top 500 uncertain points, and was often inaccurate for low-confidence cases.
• The cost of using LLMs for pairwise constraints was higher than for keyphrase expansion, but still more cost-effective than human labeling—$42 for 20K LLM labels vs. ~$41 for <700 human-labeled pairs.
• Even instruction-only prompts (without demonstrations) significantly improved clustering, suggesting that natural language instructions alone can guide LLMs effectively.
• PCKMeans with LLM-generated constraints achieved a Macro F1 of 0.867 on OPIEC59k, approaching the performance of a true oracle with only 20K LLM queries.

Figure 2: We expand document representations by concatenating them with keyphrase embeddings. The keyphrases are generated by a large language model.