[Paper Review] Mining Frequent Itemsets over Uncertain Databases

Yongxin Tong, Lei Chen+2 more|arXiv (Cornell University)|Aug 1, 2012

Data Mining Algorithms and Applications29 references4 citations

TL;DR

This paper provides a comprehensive experimental comparison of frequent itemset mining algorithms over uncertain databases, clarifying the relationship between two definitions—expected support-based and probabilistic frequent itemsets. It demonstrates that under large databases, these definitions are tightly connected via the Lyapunov Central Limit Theorem, enabling efficient approximation using normal distribution, and shows that approximation-based algorithms significantly outperform exact methods in speed and memory efficiency.

ABSTRACT

In recent years, due to the wide applications of uncertain data, mining frequent itemsets over uncertain databases has attracted much attention. In uncertain databases, the support of an itemset is a random variable instead of a fixed occurrence counting of this itemset. Thus, unlike the corresponding problem in deterministic databases where the frequent itemset has a unique definition, the frequent itemset under uncertain environments has two different definitions so far. The first definition, referred as the expected support-based frequent itemset, employs the expectation of the support of an itemset to measure whether this itemset is frequent. The second definition, referred as the probabilistic frequent itemset, uses the probability of the support of an itemset to measure its frequency. Thus, existing work on mining frequent itemsets over uncertain databases is divided into two different groups and no study is conducted to comprehensively compare the two different definitions. In addition, since no uniform experimental platform exists, current solutions for the same definition even generate inconsistent results. In this paper, we firstly aim to clarify the relationship between the two different definitions. Through extensive experiments, we verify that the two definitions have a tight connection and can be unified together when the size of data is large enough. Secondly, we provide baseline implementations of eight existing representative algorithms and test their performances with uniform measures fairly. Finally, according to the fair tests over many different benchmark data sets, we clarify several existing inconsistent conclusions and discuss some new findings.

Motivation & Objective

• To clarify the theoretical and empirical relationship between two existing definitions of frequent itemsets in uncertain databases: expected support-based and probabilistic frequent itemsets.
• To provide fair, uniform performance evaluation of eight representative algorithms under consistent experimental conditions.
• To resolve inconsistencies in prior literature by testing algorithms on diverse benchmark datasets with standardized metrics.
• To identify and validate new performance patterns and efficiency trade-offs in exact and approximate algorithms for uncertain data.

Proposed method

• Proposes baseline implementations of eight representative algorithms for both expected support-based and probabilistic frequent itemset mining.
• Employs uniform evaluation metrics (running time, memory usage) across multiple benchmark datasets to ensure fair comparison.
• Applies the Lyapunov Central Limit Theorem to justify approximating the Poisson Binomial distribution of itemset support with a normal distribution for large databases.
• Uses Chernoff bounds to filter out infrequent itemsets early in exact probabilistic algorithms, improving efficiency.
• Employs cumulative distribution function (CDF) approximations for frequent probability calculation based on expected support and variance.
• Validates findings through extensive experiments on synthetic and real-world uncertain databases with varying density and minimum support thresholds.

Experimental results

Research questions

• RQ1 How are the expected support-based and probabilistic frequent itemset definitions related in uncertain databases?
• RQ2 Do existing algorithms produce consistent performance rankings across different datasets and configurations?
• RQ3 Can the probabilistic frequent itemset mining problem be efficiently solved using approximations based on the normal distribution?
• RQ4 What are the relative performance trade-offs between exact and approximate algorithms in terms of speed and memory usage?
• RQ5 Under what conditions do approximation algorithms achieve near-perfect accuracy, and why?

Key findings

• The expected support-based and probabilistic frequent itemset definitions are tightly connected when the number of transactions exceeds 10,000, enabling the use of normal distribution approximation.
• Approximation-based probabilistic frequent itemset mining algorithms achieve near-100% frequent probability (≈1) for most itemsets in large databases, validating the theoretical approximation.
• Normal distribution-based algorithms significantly outperform Poisson-based counterparts in both speed and memory efficiency.
• For expected support-based mining, UApriori is fastest in dense databases with high min sup, while UH-Mine excels in sparse or low-min sup settings.
• Among exact probabilistic algorithms, DC is fastest in most cases but incurs high memory cost due to recursive result storage; DP is faster than DC when conditions allow.
• Approximation algorithms consistently outperform exact algorithms in efficiency and memory usage, suggesting that existing expected support-based solutions can be extended to probabilistic mining by computing variance.