[Paper Review] Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting

Nicholas C. Harris, Anand Butani+1 more|arXiv (Cornell University)|Apr 18, 2024

Topic Modeling4 citations

TL;DR

This paper proposes enhancing text embedding performance by using GPT-3.5 to enrich and rewrite input text before embedding, improving grammar, context, and terminology. On the TwitterSemEval 2015 dataset, the best prompt achieved a cosine similarity score of 85.34, surpassing the previous SOTA on the MTEB leaderboard by 8.21 points.

ABSTRACT

Embedding models are crucial for various natural language processing tasks but can be limited by factors such as limited vocabulary, lack of context, and grammatical errors. This paper proposes a novel approach to improve embedding performance by leveraging large language models (LLMs) to enrich and rewrite input text before the embedding process. By utilizing ChatGPT 3.5 to provide additional context, correct inaccuracies, and incorporate metadata, the proposed method aims to enhance the utility and accuracy of embedding models. The effectiveness of this approach is evaluated on three datasets: Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual Classification. Results demonstrate significant improvements over the baseline model on the TwitterSemEval 2015 dataset, with the best-performing prompt achieving a score of 85.34 compared to the previous best of 81.52 on the Massive Text Embedding Benchmark (MTEB) Leaderboard. However, performance on the other two datasets was less impressive, highlighting the importance of considering domain-specific characteristics. The findings suggest that LLM-based text enrichment has shown promising results to improve embedding performance, particularly in certain domains. Hence, numerous limitations in the process of embedding can be avoided.

Motivation & Objective

• To address limitations in text embedding models, such as poor context handling, grammatical sensitivity, and limited vocabulary, by leveraging LLMs for pre-processing.
• To investigate whether LLM-based text enrichment and rewriting can improve downstream NLP performance without fine-tuning the embedding model itself.
• To evaluate the effectiveness of different prompt engineering strategies in enhancing embedding quality across diverse NLP datasets.
• To determine whether LLM-augmented text preprocessing can outperform strong baseline embedding models on benchmark tasks.

Proposed method

• Utilize GPT-3.5 to rewrite and enrich raw input text by correcting grammar, resolving ambiguities, expanding acronyms, and adding contextual metadata.
• Apply four distinct prompting strategies—ranging from general instructions to task-specific, concise, and optimized prompts—to guide the LLM in text enhancement.
• Feed the enriched text into the text-embedding-3-large model for vectorization, without any fine-tuning of the embedding model.
• Evaluate performance using standard benchmarks: MTEB for TwitterSemEval 2015, Banking77Classification, and Amazon Counter-factual Classification.
• Compare results across prompt variations and baseline models, including text-embedding-3-large and SFR-Embedding-Mistral.
• Conduct qualitative analysis of LLM-generated text to assess improvements in coherence, informativeness, and semantic clarity.

Experimental results

Research questions

• RQ1 Can LLM-based text enrichment significantly improve the performance of pre-trained text embedding models on downstream NLP tasks?
• RQ2 How do different prompting strategies for LLM-based text rewriting affect the quality of resulting embeddings?
• RQ3 Does LLM-based preprocessing outperform strong baseline embedding models like text-embedding-3-large and SFR-Embedding-Mistral on benchmark datasets?
• RQ4 In what ways does LLM-enhanced text improve semantic richness and reduce ambiguity compared to raw input?
• RQ5 Why does the method show strong gains on some datasets (e.g., TwitterSemEval 2015) but not others (e.g., Banking77, Amazon Counter-factual)?

Key findings

• The best-performing prompt (Prompt 4) achieved a cosine similarity score of 85.34 on the TwitterSemEval 2015 dataset, significantly outperforming the baseline text-embedding-3-large model (77.13) by 8.21 points.
• Prompt 4 also surpassed the leading model on the MTEB Leaderboard, SFR-Embedding-Mistral, on the TwitterSemEval 2015 benchmark.
• On the Banking77Classification dataset, the best prompt (Prompt 4) achieved 79.71% accuracy, which was still below the baseline model’s performance.
• On the Amazon Counter-factual Classification dataset, the best prompt (Prompt 4) achieved 68.0% accuracy, falling short of the baseline model’s performance.
• Qualitative analysis confirmed that GPT-3.5 effectively improved text by correcting errors, disambiguating polysemous terms, expanding acronyms, and adding relevant context and metadata.
• The study demonstrates that LLM-based text enrichment can substantially improve embedding performance in specific domains, particularly where text is informal or ambiguous, such as social media text.