[Paper Review] Surpassing Cosine Similarity for Multidimensional Comparisons: Dimension Insensitive Euclidean Metric

Federico Tessari, Yao, Kunpeng+1 more|arXiv (Cornell University)|Jul 11, 2024

Face and Expression Recognition4 citations

TL;DR

This paper introduces the Dimension Insensitive Euclidean Metric (DIEM), a novel similarity measure that overcomes the dimensionality bias inherent in cosine similarity by normalizing vector length and direction in a way that preserves interpretability across high-dimensional spaces. DIEM demonstrates superior robustness and consistency in multidimensional comparisons, outperforming cosine similarity in both theoretical analysis and application to large language models.

ABSTRACT

Advances in computational power and hardware efficiency have enabled tackling increasingly complex, high-dimensional problems. While artificial intelligence (AI) achieves remarkable results, the interpretability of high-dimensional solutions remains challenging. A critical issue is the comparison of multidimensional quantities, essential in techniques like Principal Component Analysis. Metrics such as cosine similarity are often used, for example in the development of natural language processing algorithms or recommender systems. However, the interpretability of such metrics diminishes as dimensions increase. This paper analyzes the effects of dimensionality, revealing significant limitations of cosine similarity, particularly its dependency on the dimension of vectors, leading to biased and poorly interpretable outcomes. To address this, we introduce a Dimension Insensitive Euclidean Metric (DIEM) which demonstrates superior robustness and generalizability across dimensions. DIEM maintains consistent variability and eliminates the biases observed in traditional metrics, making it a reliable tool for high-dimensional comparisons. An example of the advantages of DIEM over cosine similarity is reported for a large language model application. This novel metric has the potential to replace cosine similarity, providing a more accurate and insightful method to analyze multidimensional data in fields ranging from neuromotor control to machine learning.

Motivation & Objective

• To address the growing challenge of interpreting high-dimensional data in machine learning and AI applications.
• To identify and resolve the dimensionality bias in traditional similarity metrics like cosine similarity.
• To develop a robust, generalizable metric that maintains consistent interpretability regardless of vector dimensionality.
• To provide a reliable alternative to cosine similarity for use in principal component analysis, NLP, and recommender systems.
• To validate the new metric through theoretical analysis and real-world application in large language models.

Proposed method

• The DIEM metric is derived by normalizing the Euclidean distance between vectors using a dimension-invariant scaling factor that accounts for the expected length of random vectors in high-dimensional space.
• It redefines similarity by combining normalized vector magnitude and angular difference, ensuring that the metric remains stable across different dimensionalities.
• The method introduces a correction term based on the chi distribution, which models the expected norm of standard normal vectors in d dimensions, thereby removing dimension-dependent bias.
• DIEM is designed to be computationally efficient and compatible with existing machine learning pipelines, requiring minimal modification to integrate.
• The metric is analytically derived and validated through theoretical bounds and empirical comparisons across synthetic and real-world datasets.
• The approach is evaluated using a large language model application, demonstrating improved consistency and interpretability in embedding comparisons.

Experimental results

Research questions

• RQ1 How does the performance of cosine similarity degrade with increasing vector dimensionality?
• RQ2 Can a similarity metric be designed that maintains consistent interpretability across varying dimensionalities?
• RQ3 What mathematical transformation can eliminate dimensionality bias in vector similarity measures?
• RQ4 How does DIEM compare to cosine similarity in high-dimensional embedding spaces, particularly in NLP tasks?
• RQ5 To what extent does DIEM improve the reliability of multidimensional comparisons in real-world machine learning applications?

Key findings

• DIEM eliminates the dimensionality bias present in cosine similarity, which systematically underestimates similarity in high-dimensional spaces.
• The metric maintains consistent variability across dimensions, ensuring stable and interpretable similarity scores regardless of vector length or dimension count.
• In a large language model application, DIEM demonstrated more reliable and consistent embedding comparisons than cosine similarity.
• Theoretical analysis confirms that DIEM’s expected value remains invariant under dimension changes, unlike cosine similarity.
• DIEM outperforms cosine similarity in both synthetic experiments and real-world NLP tasks, showing improved robustness and generalizability.
• The correction factor based on the chi distribution effectively normalizes vector norms, removing the primary source of bias in high-dimensional similarity measures.