[Paper Review] Stability of similarity measurements for bipartite networks

Jian-Guo Liu, Lei Hou+3 more|arXiv (Cornell University)|Nov 30, 2015

Complex Network Analysis Techniques35 references3 citations

TL;DR

This paper investigates the stability of 15 similarity measurements in bipartite networks, proposing a framework to classify them based on stability across random data splits. It finds that similarities cluster into three groups with distinct mathematical structures, and demonstrates that using only stable similarities significantly improves recommendation system reliability by filtering out false information.

ABSTRACT

Similarity is a fundamental measure in network analyses and machine learning algorithms, with wide applications ranging from personalized recommendation to socio-economic dynamics. We argue that an effective similarity measurement should guarantee the stability even under some information loss. With six bipartite networks, we investigate the stabilities of fifteen similarity measurements by comparing the similarity matrixes of two data samples which are randomly divided from original data sets. Results show that, the fifteen measurements can be well classified into three clusters according to their stabilities, and measurements in the same cluster have similar mathematical definitions. In addition, we develop a top-$n$-stability method for personalized recommendation, and find that the unstable similarities would recommend false information to users, and the performance of recommendation would be largely improved by using stable similarity measurements. This work provides a novel dimension to analyze and evaluate similarity measurements, which can further find applications in link prediction, personalized recommendation, clustering algorithms, community detection and so on.

Motivation & Objective

• To evaluate the stability of 15 widely used similarity measurements in user-object bipartite networks.
• To determine whether similarity measures produce consistent results across different data samples, given that object properties are inherently unchangeable.
• To classify similarity measures based on their stability behavior and identify underlying mathematical patterns.
• To investigate how similarity stability affects recommendation performance in real-world systems.
• To propose a top-\(n\)-stability method that improves recommendation stability by filtering out unreliable similarities.

Proposed method

• The authors split each of six real-world bipartite networks into two random sub-samples to compute similarity matrices, then compared the resulting matrices using correlation-based stability metrics.
• Stability is quantified by computing the Pearson correlation coefficient between similarity matrices derived from two independent data samples.
• Similarity measurements are grouped into clusters based on their stability profiles using hierarchical clustering on stability scores.
• The top-\(n\)-stability method selects only the most stable similarities (e.g., top \(n\) most stable object pairs) for downstream recommendation tasks.
• The method evaluates recommendation performance using standard metrics (e.g., precision, recall) on both full and stable-only similarity inputs.
• Core similarity formulas include cosine similarity (COS), Pearson correlation (PC), common neighbors (CN), and various normalized variants like Jaccard (JAC), Salton (SAL), and resource allocation (RA).

Experimental results

Research questions

• RQ1 How stable are 15 common similarity measurements when computed on different random sub-samples of the same bipartite network?
• RQ2 Can similarity measurements be systematically classified based on their stability behavior?
• RQ3 What mathematical or structural properties underlie the stability patterns observed in similarity measurements?
• RQ4 To what extent does using only stable similarities improve the reliability of recommendation systems?
• RQ5 Can a top-\(n\)-stability filtering method effectively enhance recommendation stability by removing unstable, potentially misleading similarities?

Key findings

• The 15 similarity measurements are well clustered into three distinct groups based on their stability profiles, with each group sharing similar mathematical formulations.
• Similarity measures based on the same underlying principle—such as those normalized by the minimum or maximum popularity—tend to exhibit similar stability behavior.
• The top-\(n\)-stability method significantly improves recommendation system stability by filtering out unstable, false information from similarity matrices.
• Unstable similarities were found to be a major source of noise and error in recommendations, and their removal leads to more reliable and consistent prediction outcomes.
• The study demonstrates that similarity stability is a critical but often overlooked factor in link prediction and recommendation systems, especially in sparse and evolving networks.
• The stability of a similarity measure is not solely determined by its formula but also by the underlying network structure and data sampling variability.