[Paper Review] Beauty and Brains: Detecting Anomalous Pattern Co-Occurrences

Roel Bertens, Jilles Vreeken+1 more|arXiv (Cornell University)|Dec 22, 2015

Data Visualization and Analytics21 references3 citations

TL;DR

This paper introduces a novel class of anomalies in transaction data—unexpected co-occurrences of patterns that are individually common but rarely found together. The authors propose the Beauty and Brains (BnB) score, which identifies such anomalies by measuring the maximum novelty (negative log lift) of pattern co-occurrences in a transaction, enabling fast detection and interpretable explanations without mining all association rules.

ABSTRACT

Our world is filled with both beautiful and brainy people, but how often does a Nobel Prize winner also wins a beauty pageant? Let us assume that someone who is both very beautiful and very smart is more rare than what we would expect from the combination of the number of beautiful and brainy people. Of course there will still always be some individuals that defy this stereotype; these beautiful brainy people are exactly the class of anomaly we focus on in this paper. They do not posses intrinsically rare qualities, it is the unexpected combination of factors that makes them stand out. In this paper we define the above described class of anomaly and propose a method to quickly identify them in transaction data. Further, as we take a pattern set based approach, our method readily explains why a transaction is anomalous. The effectiveness of our method is thoroughly verified with a wide range of experiments on both real world and synthetic data.

Motivation & Objective

• To define and formalize a new class of anomalies in transaction data: unexpected co-occurrences of patterns that are individually frequent but rarely paired.
• To address the gap in existing anomaly detection methods, which fail to identify anomalies arising from rare co-occurrences of common patterns.
• To develop an efficient algorithm that computes anomaly scores without exhaustive mining of all association rules.
• To provide interpretable explanations by identifying the specific pattern co-occurrence responsible for a transaction's anomaly score.
• To demonstrate the method’s effectiveness on synthetic and real-world datasets, showing it detects anomalies missed by state-of-the-art techniques.

Proposed method

• Proposes the BnB score as the negative logarithm of the maximum lift among all pattern co-occurrences present in a transaction.
• Uses a Slim pattern set to reduce computational cost, avoiding the need to mine all closed frequent itemsets.
• Applies the BnB score to rank transactions based on the most surprising (least expected) co-occurrence of patterns.
• Employs a bootstrap-based significance test to assess whether an anomaly is statistically meaningful.
• Integrates the BnB score with existing anomaly detection frameworks to complement, not replace, other anomaly classes.
• Uses the method to explain anomalies by identifying the specific pattern pair with the highest novelty (lowest lift) in a transaction.

Experimental results

Research questions

• RQ1 Can a new class of anomalies be formally defined as unexpected co-occurrences of individually common patterns in transaction data?
• RQ2 Can an efficient and scalable method detect such anomalies without exhaustive mining of all association rules?
• RQ3 Does the BnB score effectively rank synthetic anomalies planted in data higher than normal transactions?
• RQ4 Can the BnB method detect real-world anomalies that are missed by existing methods like OC3 and CompreX?
• RQ5 Can the BnB score provide interpretable explanations by identifying the specific pattern co-occurrence responsible for a transaction’s anomaly?

Key findings

• The BnB score successfully identified all planted class 2 anomalies (unexpected pattern co-occurrences) at the top of the ranking in synthetic datasets.
• On the Adult dataset, the BnB method detected a highly anomalous individual described as a 'female husband', which was not flagged by OC3 or CompreX.
• In the Zoo dataset, the platypus—a unique oviparous mammal—had less than an 8% chance of being considered 'normal' under the data distribution, indicating a significant anomaly.
• The BnB method detected a day in the Bike Sharing dataset with a rare co-occurrence of low real temperature and high apparent temperature, a class 2 anomaly missed by prior methods.
• On the Mammals dataset, BnB correctly identified two top-ranked areas where two animal groups' habitats intersected only in a single small region, explaining the anomaly via pattern co-occurrence.
• Using a Slim pattern set, the BnB method achieved comparable results to a full closed frequent itemset approach but in a fraction of the time, demonstrating scalability.