[Paper Review] Massively Parallel and Asynchronous Tsetlin Machine Architecture Supporting Almost Constant-Time Scaling

K. Darshana Abeyrathna, Bimal Bhattarai+6 more|arXiv (Cornell University)|Sep 10, 2020

Algorithms and Data CompressionComputer Science26 references21 citations

TL;DR

This paper proposes a massively parallel, asynchronous Tsetlin Machine architecture that eliminates synchronization bottlenecks by using local voting tallies per clause, enabling nearly constant-time training scaling across 20 to 7,000 clauses on a GPU. The method achieves up to 50× faster learning with minimal accuracy loss, even when operating on outdated voting data, due to robust decentralized learning via thread-level parallelism and atomic updates.

ABSTRACT

Using logical clauses to represent patterns, Tsetlin Machines (TMs) have recently obtained competitive performance in terms of accuracy, memory footprint, energy, and learning speed on several benchmarks. Each TM clause votes for or against a particular class, with classification resolved using a majority vote. While the evaluation of clauses is fast, being based on binary operators, the voting makes it necessary to synchronize the clause evaluation, impeding parallelization. In this paper, we propose a novel scheme for desynchronizing the evaluation of clauses, eliminating the voting bottleneck. In brief, every clause runs in its own thread for massive native parallelism. For each training example, we keep track of the class votes obtained from the clauses in local voting tallies. The local voting tallies allow us to detach the processing of each clause from the rest of the clauses, supporting decentralized learning. This means that the TM most of the time will operate on outdated voting tallies. We evaluated the proposed parallelization across diverse learning tasks and it turns out that our decentralized TM learning algorithm copes well with working on outdated data, resulting in no significant loss in learning accuracy. Furthermore, we show that the proposed approach provides up to 50 times faster learning. Finally, learning time is almost constant for reasonable clause amounts (employing from 20 to 7,000 clauses on a Tesla V100 GPU). For sufficiently large clause numbers, computation time increases approximately proportionally. Our parallel and asynchronous architecture thus allows processing of massive datasets and operating with more clauses for higher accuracy.

Motivation & Objective

• To overcome the synchronization bottleneck in traditional Tsetlin Machine learning, which limits parallelization and scalability.
• To enable massive native parallelism by allowing each clause to operate independently in its own thread.
• To decouple clause processing from global voting synchronization using local voting tallies, supporting decentralized learning.
• To evaluate the robustness of learning when operating on outdated or partially computed voting data.
• To achieve near-constant training time scaling across a wide range of clause counts (20–7,000) on GPU hardware.

Proposed method

• Each clause runs in its own thread, enabling massive native parallelism across GPU cores.
• Local voting tallies are maintained per training example to track clause outputs independently, decoupling clause processing from global synchronization.
• Clause updates are performed asynchronously using atomic operations on voting tallies, minimizing coordination overhead.
• The system operates on outdated voting data most of the time, simulating a decentralized learning regime with minimal coordination.
• The architecture supports both training and inference with minimal memory overhead, adding only 1 bit per example per clause for tallying.
• The method is evaluated on diverse benchmarks including image classification, word sense disambiguation, and regression tasks using a Tesla V100 GPU.

Experimental results

Research questions

• RQ1 Can Tsetlin Machine learning be scaled efficiently using massive parallelism without global synchronization?
• RQ2 How does learning performance degrade when clauses operate on outdated voting tallies in an asynchronous setting?
• RQ3 Does the proposed architecture achieve near-constant training time scaling across varying numbers of clauses?
• RQ4 Can the system maintain high accuracy while achieving up to 50× speedup in learning time?
• RQ5 How well does the decentralized, asynchronous learning mechanism perform on diverse machine learning tasks?

Key findings

• The proposed architecture achieves up to 50× faster learning compared to conventional Tsetlin Machines, with no significant loss in accuracy.
• Training time remains nearly constant for clause counts between 20 and 7,000 on a Tesla V100 GPU, demonstrating almost ideal scaling.
• For clause counts beyond 7,000, computation time increases proportionally, indicating that the GPU’s 5,120 cores are fully exploited up to that point.
• The system maintains high accuracy even when clauses operate on outdated voting data, demonstrating strong robustness to desynchronization.
• The method enables efficient decentralized learning, with only atomic operations on local voting tallies required for coordination.
• Empirical evaluation on word sense disambiguation and image classification tasks confirms competitive accuracy, with F1 scores of 97.53% on JAVA and 95.1% on APPLE datasets.