[Paper Review] Distributed Word Representation in Tsetlin Machine.

Rohan Kumar Yadav, Lei Jiao+2 more|arXiv (Cornell University)|Apr 14, 2021

Multimodal Machine Learning Applications22 references4 citations

TL;DR

This paper proposes a novel method to integrate pre-trained word embeddings like word2vec and GloVe into the Tsetlin Machine (TM) for improved NLP performance while preserving interpretability. By extracting semantically related words from pre-trained vectors as input features, the approach significantly boosts TM accuracy, achieving performance on par with deep neural networks in tasks like sentiment analysis and Word Sense Disambiguation.

ABSTRACT

Tsetlin Machine (TM) is an interpretable pattern recognition algorithm based on propositional logic. The algorithm has demonstrated competitive performance in many Natural Language Processing (NLP) tasks, including sentiment analysis, text classification, and Word Sense Disambiguation (WSD). To obtain human-level interpretability, legacy TM employs Boolean input features such as bag-of-words (BOW). However, the BOW representation makes it difficult to use any pre-trained information, for instance, word2vec and GloVe word representations. This restriction has constrained the performance of TM compared to deep neural networks (DNNs) in NLP. To reduce the performance gap, in this paper, we propose a novel way of using pre-trained word representations for TM. The approach significantly enhances the TM performance and maintains interpretability at the same time. We achieve this by extracting semantically related words from pre-trained word representations as input features to the TM. Our experiments show that the accuracy of the proposed approach is significantly higher than the previous BOW-based TM, reaching the level of DNN-based models.

Motivation & Objective

• To overcome the performance limitations of traditional Tsetlin Machines that rely on bag-of-words (BOW) features in NLP tasks.
• To enable the use of pre-trained word representations such as word2vec and GloVe within the interpretable framework of the Tsetlin Machine.
• To maintain the model's interpretability while significantly improving its accuracy in NLP tasks.
• To close the performance gap between Tsetlin Machines and deep neural networks in NLP applications.

Proposed method

• Extracting semantically related words from pre-trained word embeddings (e.g., word2vec, GloVe) to form input features for the Tsetlin Machine.
• Transforming dense vector representations into discrete, semantically meaningful features suitable for propositional logic-based learning.
• Using the Tsetlin Machine’s inherent ability to learn from Boolean features to classify text based on these derived semantic features.
• Preserving interpretability by avoiding end-to-end learning and retaining the logical structure of the Tsetlin Machine.
• Applying the resulting feature set to standard NLP tasks such as sentiment analysis and Word Sense Disambiguation.

Experimental results

Research questions

• RQ1 Can pre-trained word embeddings be effectively integrated into the Tsetlin Machine without sacrificing interpretability?
• RQ2 How does the performance of the Tsetlin Machine improve when using semantic features derived from pre-trained embeddings compared to traditional bag-of-words features?
• RQ3 To what extent can the enhanced Tsetlin Machine match the accuracy of deep neural network models in NLP tasks?

Key findings

• The proposed method achieves significantly higher accuracy than the baseline BOW-based Tsetlin Machine across multiple NLP tasks.
• The performance of the enhanced Tsetlin Machine reaches levels comparable to those of deep neural network-based models in sentiment analysis and Word Sense Disambiguation.
• The integration of pre-trained word representations improves the semantic expressiveness of the input features while maintaining the model’s interpretability.
• The method successfully bridges the performance gap between interpretable models and state-of-the-art deep learning models in NLP.