[Paper Review] LLMs are Also Effective Embedding Models: An In-depth Overview

Chongyang Tao, Tao Shen+7 more|arXiv (Cornell University)|Dec 17, 2024

Financial Distress and Bankruptcy Prediction4 citations

TL;DR

This survey establishes that large language models (LLMs) are highly effective as embedding models, outperforming traditional encoder-only models like BERT through two main approaches: direct prompting with optimized prompt designs and data-centric fine-tuning involving architecture, training objectives, and data construction. The key contribution is a comprehensive framework for leveraging LLMs as dense, general-purpose embeddings with strong performance across retrieval, classification, and RAG tasks.

ABSTRACT

Large language models (LLMs) have revolutionized natural language processing by achieving state-of-the-art performance across various tasks. Recently, their effectiveness as embedding models has gained attention, marking a paradigm shift from traditional encoder-only models like ELMo and BERT to decoder-only, large-scale LLMs such as GPT, LLaMA, and Mistral. This survey provides an in-depth overview of this transition, beginning with foundational techniques before the LLM era, followed by LLM-based embedding models through two main strategies to derive embeddings from LLMs. 1) Direct prompting: We mainly discuss the prompt designs and the underlying rationale for deriving competitive embeddings. 2) Data-centric tuning: We cover extensive aspects that affect tuning an embedding model, including model architecture, training objectives, data constructions, etc. Upon the above, we also cover advanced methods for producing embeddings from longer texts, multilingual, code, cross-modal data, as well as reasoning-aware and other domain-specific scenarios. Furthermore, we discuss factors affecting choices of embedding models, such as performance/efficiency comparisons, dense vs sparse embeddings, pooling strategies, and scaling law. Lastly, the survey highlights the limitations and challenges in adapting LLMs for embeddings, including cross-task embedding quality, trade-offs between efficiency and accuracy, low-resource, long-context, data bias, robustness, etc. This survey serves as a valuable resource for researchers and practitioners by synthesizing current advancements, highlighting key challenges, and offering a comprehensive framework for future work aimed at enhancing the effectiveness and efficiency of LLMs as embedding models.

Motivation & Objective

• To analyze the paradigm shift from encoder-only models (e.g., BERT) to decoder-only LLMs (e.g., GPT, LLaMA) as effective embedding models.
• To systematically compare tuning-free and fine-tuned LLM-based embedding methods, focusing on prompt design and data-centric adaptation strategies.
• To evaluate the trade-offs between efficiency, accuracy, and robustness in LLM-based embeddings across diverse settings.
• To identify open challenges such as low-resource adaptation, adversarial vulnerability, and misalignment between pre-training and embedding objectives.

Proposed method

• Categorizes LLM-based embedding methods into two main strategies: direct prompting and data-centric tuning.
• Analyzes prompt engineering techniques, including instruction tuning and contrastive prompting, to derive high-quality embeddings without fine-tuning.
• Reviews model architecture choices, loss functions (e.g., contrastive loss), and data construction methods (e.g., synthetic data, retrieval-augmented data) for effective fine-tuning.
• Examines pooling strategies (e.g., mean, CLS, [CLS]) and their impact on embedding quality and downstream performance.
• Evaluates scaling laws and efficiency-accuracy trade-offs in LLM-based embedding systems.
• Discusses advanced techniques for long-context, multilingual, and cross-modal embedding generation.

Figure 1: Illustration of direct prompting or tuning LLMs as embedding models.

Experimental results

Research questions

• RQ1 How do LLMs compare to traditional encoder-only models (e.g., BERT) in generating effective semantic embeddings?
• RQ2 What are the most effective prompting strategies for deriving high-quality embeddings directly from LLMs without fine-tuning?
• RQ3 How does data-centric fine-tuning—via architecture, objective, and data curation—improve LLM-based embedding performance?
• RQ4 What are the key challenges in adapting LLM embeddings to low-resource, long-context, or multilingual domains?
• RQ5 How can LLM embeddings be made robust to adversarial attacks and distributional shifts?

Key findings

• LLMs such as GPT, LLaMA, and Mistral achieve state-of-the-art performance in embedding tasks due to their larger context modeling capacity and extensive pre-training on trillions of tokens.
• Direct prompting with carefully designed instructions can produce competitive embeddings comparable to fine-tuned models, reducing the need for costly fine-tuning.
• Data-centric tuning, including contrastive loss and curated data, significantly improves embedding quality, especially for downstream retrieval and classification tasks.
• Pooling strategies like mean pooling and [CLS]-based pooling show measurable differences in performance, with mean pooling often outperforming in generalization.
• LLMs face challenges in low-resource domains due to data scarcity and domain shift, leading to poor generalization without domain-specific fine-tuning.
• Adversarial robustness remains a critical limitation, as small input perturbations can drastically alter embeddings, threatening real-world deployment in security-sensitive applications.

Figure 3: The performance trend of models with varying sizes on MTEB benchmark, based on results from Muennighoff et al. ( 2022 ) .