[Paper Review] The Convolutional Tsetlin Machine

Ole‐Christoffer Granmo, Sondre Glimsdal+4 more|arXiv (Cornell University)|May 23, 2019

Machine Learning and AlgorithmsComputer Science43 references48 citations

TL;DR

The Convolutional Tsetlin Machine (CTM) extends the interpretable Tsetlin Machine to image data by using clause-based convolution filters with location-aware patches, achieving competitive accuracy on MNIST, Kuzushiji-MNIST, Fashion-MNIST, and 2D Noisy XOR.

ABSTRACT

Convolutional neural networks (CNNs) have obtained astounding successes for important pattern recognition tasks, but they suffer from high computational complexity and the lack of interpretability. The recent Tsetlin Machine (TM) attempts to address this lack by using easy-to-interpret conjunctive clauses in propositional logic to solve complex pattern recognition problems. The TM provides competitive accuracy in several benchmarks, while keeping the important property of interpretability. It further facilitates hardware-near implementation since inputs, patterns, and outputs are expressed as bits, while recognition and learning rely on straightforward bit manipulation. In this paper, we exploit the TM paradigm by introducing the Convolutional Tsetlin Machine (CTM), as an interpretable alternative to CNNs. Whereas the TM categorizes an image by employing each clause once to the whole image, the CTM uses each clause as a convolution filter. That is, a clause is evaluated multiple times, once per image patch taking part in the convolution. To make the clauses location-aware, each patch is further augmented with its coordinates within the image. The output of a convolution clause is obtained simply by ORing the outcome of evaluating the clause on each patch. In the learning phase of the TM, clauses that evaluate to 1 are contrasted against the input. For the CTM, we instead contrast against one of the patches, randomly selected among the patches that made the clause evaluate to 1. Accordingly, the standard Type I and Type II feedback of the classic TM can be employed directly, without further modification. The CTM obtains a peak test accuracy of 99.4% on MNIST, 96.31% on Kuzushiji-MNIST, 91.5% on Fashion-MNIST, and 100.0% on the 2D Noisy XOR Problem, which is competitive with results reported for simple 4-layer CNNs, BinaryConnect, Logistic Circuits and an FPGA-accelerated Binary CNN.

Motivation & Objective

• Introduce the Convolutional Tsetlin Machine (CTM) as an interpretable alternative to CNNs.
• Adapt the TM learning rules to operate over image patches via convolution-like filtering.
• Demonstrate CTM’s recognition and learning performance on standard benchmarks and a 2D XOR task.

Proposed method

• Represent images as binary inputs and define clause-based convolution filters of size W×W×Z×2.
• Augment each image patch with encoded location information to make clauses location-aware.
• Evaluate each clause on all patches and aggregate via OR to produce a clause output per image.
• Apply Type I and Type II feedback from the classic TM to updates of Tsetlin Automata within clauses, adapted to the CTM setting by selecting a random patch among the ones that activated the clause.
• Optionally incorporate integer clause weighting to perform a weighted majority vote across clauses.
• Demonstrate parallelizable, hardware-friendly operations due to bit-level inputs and simple bit manipulations.

Experimental results

Research questions

• RQ1 Can CTM achieve competitive accuracy for image classification while maintaining interpretability?
• RQ2 How can the TM learning feedback (Type I and Type II) be adapted to a convolutional, patch-based setting?
• RQ3 What is the impact of location-awareness and patch-wise clause outputs on recognition performance?
• RQ4 How does clause weighting affect accuracy and computational efficiency in CTM?

Key findings

• CTM attains peak test accuracies of 99.4% on MNIST, 96.31% on Kuzushiji-MNIST, 91.5% on Fashion-MNIST, and 100.0% on 2D Noisy XOR with competitive benchmarks.
• CTM operates with linear growth in computation with respect to the number of clauses and image patches and benefits from parallelizable updates.
• Incorporating location information helps the filters become location-aware patterns suitable for image tasks.
• Clause weighting further improves performance and efficiency, enabling a single weighted vote to replace multiple clauses.
• CTM demonstrates competitive results against simple CNNs, BinaryConnect, Logistic Circuits, and FPGA-accelerated Binary CNNs on selected datasets.