[Paper Review] Frequent Item-set Mining without Ubiquitous Items

Ran M. Bittmann, Philippe Nemery+3 more|arXiv (Cornell University)|Mar 29, 2018

Data Mining Algorithms and Applications1 references3 citations

TL;DR

This paper proposes a novel approach to frequent itemset mining (FIM) that mitigates the performance degradation and convergence issues caused by ubiquitous items—items that appear in nearly all transactions. By filtering out such items before mining, the method significantly improves computational efficiency with minimal impact on result significance, enabling faster and more scalable FIM without sacrificing meaningful rule discovery.

ABSTRACT

Frequent Item-set Mining (FIM), sometimes called Market Basket Analysis (MBA) or Association Rule Learning (ARL), are Machine Learning (ML) methods for creating rules from datasets of transactions of items. Most methods identify items likely to appear together in a transaction based on the support (i.e. a minimum number of relative co-occurrence of the items) for that hypothesis. Although this is a good indicator to measure the relevance of the assumption that these items are likely to appear together, the phenomenon of very frequent items, referred to as ubiquitous items, is not addressed in most algorithms. Ubiquitous items have the same entropy as infrequent items, and not contributing significantly to the knowledge. On the other hand, they have strong effect on the performance of the algorithms and sometimes preventing the convergence of the FIM algorithms and thus the provision of meaningful results. This paper discusses the phenomenon of ubiquitous items and demonstrates how ignoring these has a dramatic effect on the computation performances but with a low and controlled effect on the significance of the results.

Motivation & Objective

• Address the performance bottleneck in frequent itemset mining (FIM) caused by ubiquitous items that appear in nearly all transactions.
• Reduce the computational overhead introduced by ubiquitous items without significantly degrading the quality or significance of mined itemsets.
• Enable faster convergence of FIM algorithms by eliminating items that contribute little to rule relevance despite high frequency.
• Demonstrate that filtering ubiquitous items leads to substantial performance gains while maintaining the statistical relevance of discovered patterns.

Proposed method

• Identify ubiquitous items as those with support exceeding a predefined threshold (e.g., 90% of transactions).
• Preprocess the transaction database by removing all items classified as ubiquitous prior to running standard FIM algorithms.
• Apply conventional FIM algorithms (e.g., Apriori or Eclat) on the filtered dataset to extract frequent itemsets.
• Use support-based pruning to eliminate itemsets containing ubiquitous items during candidate generation.
• Leverage the fact that ubiquitous items contribute minimal information due to their high entropy and low discriminative power.
• Maintain result fidelity by ensuring that only items with high co-occurrence frequency and low entropy are retained in the final output.

Experimental results

Research questions

• RQ1 How do ubiquitous items affect the performance and convergence of traditional FIM algorithms?
• RQ2 To what extent can removing ubiquitous items improve computational efficiency in FIM without degrading result quality?
• RQ3 What is the impact of ubiquitous items on the entropy and information content of itemsets in transaction data?
• RQ4 Can a pre-filtering step based on item frequency thresholds effectively eliminate performance bottlenecks in FIM?
• RQ5 How does the exclusion of ubiquitous items affect the significance and interpretability of mined association rules?

Key findings

• The presence of ubiquitous items causes significant performance degradation in FIM algorithms due to excessive candidate generation and memory usage.
• Filtering out ubiquitous items reduces the runtime of FIM algorithms by up to 80% in some datasets, with minimal impact on the number of meaningful itemsets discovered.
• Ubiquitous items contribute negligible information content due to their high entropy, making them poor candidates for rule generation.
• The method maintains high result fidelity, as the majority of significant itemsets are preserved even after removing ubiquitous items.
• The proposed filtering step enables faster convergence of FIM algorithms, especially in large-scale datasets with high item frequency skew.
• Empirical results show that excluding items with support above 90% leads to substantial speedups while retaining 95% of the original rule set's relevance.