[Paper Review] Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems

Laura Caspari, Kanishka Ghosh Dastidar+3 more|arXiv (Cornell University)|Jul 11, 2024

Power Systems and Technologies4 citations

TL;DR

This paper evaluates embedding model similarity in Retrieval-Augmented Generation (RAG) systems using both representational similarity (via Centered Kernel Alignment) and functional similarity in retrieval results (via Jaccard and rank similarity). It identifies intra- and inter-family model clusters but finds high variance in retrieval similarity at low top-k values, especially for larger datasets, indicating that model choice remains non-trivial despite similarity clustering.

ABSTRACT

The choice of embedding model is a crucial step in the design of Retrieval Augmented Generation (RAG) systems. Given the sheer volume of available options, identifying clusters of similar models streamlines this model selection process. Relying solely on benchmark performance scores only allows for a weak assessment of model similarity. Thus, in this study, we evaluate the similarity of embedding models within the context of RAG systems. Our assessment is two-fold: We use Centered Kernel Alignment to compare embeddings on a pair-wise level. Additionally, as it is especially pertinent to RAG systems, we evaluate the similarity of retrieval results between these models using Jaccard and rank similarity. We compare different families of embedding models, including proprietary ones, across five datasets from the popular Benchmark Information Retrieval (BEIR). Through our experiments we identify clusters of models corresponding to model families, but interestingly, also some inter-family clusters. Furthermore, our analysis of top-k retrieval similarity reveals high-variance at low k values. We also identify possible open-source alternatives to proprietary models, with Mistral exhibiting the highest similarity to OpenAI models.

Motivation & Objective

• To address the challenge of selecting optimal embedding models for RAG systems amidst a rapidly growing number of options.
• To move beyond benchmark scores by analyzing embedding similarity in both representational and functional (retrieval) terms.
• To identify clusters of models with similar behavior to streamline model selection and enable informed alternatives to proprietary models.
• To assess whether model similarity is consistent across different datasets and retrieval top-k values.

Proposed method

• Employ Centered Kernel Alignment (CKA) to measure pairwise similarity between embeddings generated by 19 different embedding models.
• Evaluate functional similarity in RAG retrieval by computing Jaccard and rank similarity on top-k retrieved text chunks across five BEIR datasets.
• Compare both open-source and proprietary models, including OpenAI, Cohere, and Mistral-based models, across diverse data distributions.
• Use a diverse set of five BEIR benchmark datasets to test the robustness of similarity patterns across different domains and data scales.
• Analyze similarity at varying top-k values (especially low k, common in RAG) to assess stability of retrieval behavior.
• Identify model clusters via hierarchical clustering on similarity scores to guide model selection and alternative model discovery.

Figure 1. Mean CKA similarity across all five datasets. Models tend to be most similar to models belonging to their own family, though some interesting inter-family patterns are visible as well.

Experimental results

Research questions

• RQ1 How similar are the embeddings generated by different embedding models, both representationally and functionally in retrieval tasks?
• RQ2 Do models cluster into families based on architecture or origin, and are there notable inter-family similarities?
• RQ3 How does retrieval similarity vary with top-k values, particularly at low k, which are commonly used in RAG systems?
• RQ4 Can open-source models be identified as strong alternatives to proprietary models like those from OpenAI or Cohere?
• RQ5 Does model similarity vary significantly across different datasets, and is it influenced by dataset size or domain?

Key findings

• Embedding models form clear intra-family clusters, with models from the same family (e.g., BGE, UAE, mxbai) showing high similarity in both CKA and retrieval results.
• Inter-family clusters exist, with the bge-large-en-v1.5, UAE-Large-V1, and mxbai-embed-large-v1 forming a prominent cluster showing high similarity even at low top-k values.
• Retrieval similarity exhibits high variance at low top-k values (e.g., k=1 to 5), especially on larger datasets, indicating that retrieved chunks can be nearly entirely distinct across models.
• SFR-Embedding-Mistral shows the highest similarity to OpenAI’s models in terms of retrieval behavior, though Jaccard similarity at top-10 remains low on larger datasets.
• Cohere’s embedding model shows inconsistent similarity across datasets at low k, with no single open-source model emerging as a consistent alternative.
• Top-10 retrieval results across most model pairs show low to moderate Jaccard similarity, indicating that retrieved context for LLMs can vary significantly despite similar embedding representations.

Figure 2. Rank similarity over all $k$ on NFCorpus, comparing gte-large to all other models. Scores are highest and vary most for small $k$ , but then drop quickly before stabilizing for larger $k$ .