[Paper Review] An Optimized Toolbox for Advanced Image Processing with Tsetlin Machine Composites

Ylva Grønningsæter, Halvor S. Smørvik+1 more|arXiv (Cornell University)|Jun 2, 2024

Brain Tumor Detection and ClassificationNeuroscience3 citations

TL;DR

This paper introduces a novel toolbox of 22 Tsetlin Machine (TM) Specialists that leverage advanced image processing techniques—such as Canny edge detection, histogram of oriented gradients, and color thermometers—combined with a rigorous hyperparameter search via Optuna. The resulting TM Composites architecture achieves a new state-of-the-art accuracy of 82.8% on CIFAR-10, significantly outperforming prior TM-based methods by 7.7 percentage points.

ABSTRACT

The Tsetlin Machine (TM) has achieved competitive results on several image classification benchmarks, including MNIST, K-MNIST, F-MNIST, and CIFAR-2. However, color image classification is arguably still in its infancy for TMs, with CIFAR-10 being a focal point for tracking progress. Over the past few years, TM's CIFAR-10 accuracy has increased from around 61% in 2020 to 75.1% in 2023 with the introduction of Drop Clause. In this paper, we leverage the recently proposed TM Composites architecture and introduce a range of TM Specialists that use various image processing techniques. These include Canny edge detection, Histogram of Oriented Gradients, adaptive mean thresholding, adaptive Gaussian thresholding, Otsu's thresholding, color thermometers, and adaptive color thermometers. In addition, we conduct a rigorous hyperparameter search, where we uncover optimal hyperparameters for several of the TM Specialists. The result is a toolbox that provides new state-of-the-art results on CIFAR-10 for TMs with an accuracy of 82.8%. In conclusion, our toolbox of TM Specialists forms a foundation for new TM applications and a landmark for further research on TM Composites in image analysis.

Motivation & Objective

• To enhance the performance of Tsetlin Machines in color image classification, particularly on CIFAR-10, which remains underexplored despite progress in simpler benchmarks.
• To develop a modular, plug-and-play toolbox of TM Specialists using diverse image processing techniques to improve feature representation before classification.
• To conduct a comprehensive hyperparameter search using Optuna to identify optimal configurations for each TM Specialist, maximizing accuracy and robustness.
• To evaluate the scalability and performance of TM Composites by varying the number of clauses per Specialist and analyzing accuracy trends.
• To establish a new benchmark for interpretable, low-complexity machine learning on CIFAR-10, addressing the gap between accuracy and model interpretability.

Proposed method

• The authors design 22 TM Specialists, each applying a distinct image processing technique—including Canny edge detection, Otsu’s thresholding, adaptive Gaussian thresholding, and color thermometers—on CIFAR-10 images prior to TM classification.
• Each TM Specialist is trained using the Convolutional Tsetlin Machine (CTM) architecture with a configurable number of clauses, ranging from 8,000 to 64,000, to assess scalability.
• A hyperparameter search is performed using the Optuna optimization framework to identify optimal values for the Tsetlin Machine’s parameters, such as the threshold parameter T and scaling factor s, for each specialist.
• The TM Composites architecture combines multiple trained TM Specialists via a voting mechanism, where each specialist contributes a prediction, and the final class is determined by majority vote.
• The performance of the TM Composite is evaluated across varying clause counts (8K to 64K) to analyze the impact of model capacity on accuracy and convergence.
• The method includes data augmentation in some configurations to improve generalization, with results reported on both augmented and non-augmented data.

Experimental results

Research questions

• RQ1 Can a diverse set of image processing techniques significantly improve the accuracy of Tsetlin Machines on color image classification tasks like CIFAR-10?
• RQ2 What is the optimal configuration of hyperparameters (e.g., T, s, clause count) for each TM Specialist when applied to different image processing pipelines?
• RQ3 How does the number of clauses per TM Specialist affect the overall accuracy of the TM Composite on CIFAR-10?
• RQ4 Can the TM Composites architecture outperform existing state-of-the-art TM-based models on CIFAR-10, particularly in terms of accuracy and interpretability?
• RQ5 Do certain image processing techniques, such as color thermometers or adaptive thresholding, yield consistently better performance across multiple TM Specialist configurations?

Key findings

• The TM Composite composed of 22 TM Specialists, each trained with 64,000 clauses and optimized hyperparameters, achieves a new state-of-the-art accuracy of 82.8% on CIFAR-10.
• The top-performing individual TM Specialist, the 5x5 Color Thermometer with 64,000 clauses, achieves 75.4% accuracy, surpassing the previous SOTA of 75.1%.
• The accuracy of the TM Composite increases with clause count up to 32,000 clauses (82.7%), with only a marginal gain at 64,000 clauses (82.8%), indicating diminishing returns beyond a certain capacity.
• The hyperparameter search via Optuna identified robust and effective configurations for multiple TM Specialists, with accuracy consistently improving as clause count increased, except for Histogram of Oriented Gradients and its augmented variant, which plateaued after 32,000 clauses.
• The use of color thermometers and adaptive Gaussian thresholding with optimized hyperparameters yielded higher accuracy than previously reported results in earlier TM studies.
• The TM Composites architecture successfully integrates diverse image processing pipelines into a single, high-performing ensemble, demonstrating scalability and effectiveness in boosting performance.