[Paper Review] Measuring the Novelty of Natural Language Text Using the Conjunctive Clauses of a Tsetlin Machine Text Classifier

Bimal Bhattarai, Ole‐Christoffer Granmo+1 more|arXiv (Cornell University)|Nov 17, 2020

Anomaly Detection Techniques and Applications18 references4 citations

TL;DR

This paper proposes a novel, interpretable approach to open-world text classification by leveraging the conjunctive clauses of a Tsetlin Machine (TM) to measure text novelty. By counting how many clauses match an input text, the method generates a novelty score that distinguishes known from novel classes, outperforming seven baselines on three of five datasets while providing a transparent, logic-based explanation of predictions.

ABSTRACT

Most supervised text classification approaches assume a closed world, counting on all classes being present in the data at training time. This assumption can lead to unpredictable behaviour during operation, whenever novel, previously unseen, classes appear. Although deep learning-based methods have recently been used for novelty detection, they are challenging to interpret due to their black-box nature. This paper addresses \emph{interpretable} open-world text classification, where the trained classifier must deal with novel classes during operation. To this end, we extend the recently introduced Tsetlin machine (TM) with a novelty scoring mechanism. The mechanism uses the conjunctive clauses of the TM to measure to what degree a text matches the classes covered by the training data. We demonstrate that the clauses provide a succinct interpretable description of known topics, and that our scoring mechanism makes it possible to discern novel topics from the known ones. Empirically, our TM-based approach outperforms seven other novelty detection schemes on three out of five datasets, and performs second and third best on the remaining, with the added benefit of an interpretable propositional logic-based representation.

Motivation & Objective

• To address the limitations of closed-world assumptions in supervised text classification, where novel classes lead to unpredictable behavior.
• To develop an interpretable novelty detection mechanism that can flag previously unseen classes during inference.
• To leverage the propositional logic structure of Tsetlin Machines for meaningful, human-readable pattern representation in text classification.
• To enable robust detection of novel topics in text while maintaining interpretability, overcoming the black-box nature of deep learning models.

Proposed method

• The method uses the conjunctive clauses of a trained Tsetlin Machine to represent frequent, salient patterns in known training data.
• A novelty score is computed by counting the number of clauses activated by a given input text.
• This score serves as a proxy for how well the input aligns with known classes, with lower scores indicating higher novelty.
• A threshold is applied to the novelty score to classify inputs as either known or novel.
• For improved robustness, a secondary machine learning classifier is trained on the novelty scores to distinguish known from novel classes.
• The framework is evaluated using TF-IDF vectorization, PCA, and standard preprocessing across five diverse text datasets.

Experimental results

Research questions

• RQ1 Can the conjunctive clauses of a Tsetlin Machine effectively represent known topics in a way that enables detection of novel topics?
• RQ2 How does the clause-based novelty scoring mechanism compare to established outlier and clustering-based methods in multiclass text classification?
• RQ3 To what extent does the interpretability of the Tsetlin Machine’s logic-based representation enhance the reliability of novelty detection?
• RQ4 Does the proposed method maintain high performance on datasets with overlapping vocabulary between known and novel classes?

Key findings

• The TM-based framework achieved the highest accuracy on the 20 Newsgroup and BBC Sports datasets, with 82.50% and 89.47% respectively, outperforming all other methods on these two datasets.
• On the CMU Movie and Web of Science datasets, the method ranked second and third, achieving 68.15% and 70.37% accuracy, respectively, behind only the multilayer perceptron (MLP).
• The method surpassed cluster- and outlier-based baselines such as One-class SVM, Isolation Forest, and K-Means clustering on three of the five datasets.
• The framework demonstrated superior performance on datasets with clear semantic distinctions between known and novel classes, where novelty scores were well-separated.
• On the Web of Science dataset, where known and novel classes shared many common words, the method was outperformed by distance-based methods, highlighting a challenge in vocabulary-overlap scenarios.
• The use of clause activation counts as a novelty score enabled clear clustering of known and novel samples, as visualized in Figure 3, supporting the method’s discriminative power.