[Paper Review] Algebraic Machine Learning

Fernando Martín-Maroto, Gonzalo G. de Polavieja|arXiv (Cornell University)|Mar 14, 2018

Machine Learning and Algorithms13 references4 citations

TL;DR

This paper proposes a parameter-free, discrete algebraic machine learning framework that replaces traditional function minimization with cardinal minimization—finding the smallest set of irreducible algebraic components (atoms) that represent training data. The method generalizes effectively, achieving high test accuracy without overfitting, and demonstrates a strong inverse relationship between compression (fewer atoms) and error rate, validated on digit recognition and the N-Queens Completion problem.

ABSTRACT

Machine learning algorithms use error function minimization to fit a large set of parameters in a preexisting model. However, error minimization eventually leads to a memorization of the training dataset, losing the ability to generalize to other datasets. To achieve generalization something else is needed, for example a regularization method or stopping the training when error in a validation dataset is minimal. Here we propose a different approach to learning and generalization that is parameter-free, fully discrete and that does not use function minimization. We use the training data to find an algebraic representation with minimal size and maximal freedom, explicitly expressed as a product of irreducible components. This algebraic representation is shown to directly generalize, giving high accuracy in test data, more so the smaller the representation. We prove that the number of generalizing representations can be very large and the algebra only needs to find one. We also derive and test a relationship between compression and error rate. We give results for a simple problem solved step by step, hand-written character recognition, and the Queens Completion problem as an example of unsupervised learning. As an alternative to statistical learning, algebraic learning may offer advantages in combining bottom-up and top-down information, formal concept derivation from data and large-scale parallelization.

Motivation & Objective

• To develop a parameter-free, discrete learning framework that avoids function minimization and overfitting common in statistical machine learning.
• To enable generalization through algebraic compression into minimal, irreducible components (atoms) without relying on error minimization.
• To establish a theoretical and empirical link between compression (number of atoms) and test error rate in learning.
• To demonstrate the method’s effectiveness on supervised and unsupervised problems, including handwritten digit classification and the N-Queens Completion problem.
• To explore symmetry exploitation in learned atoms to generate new solutions without additional training.

Proposed method

• The method uses an extended semilattice algebra to represent data and relationships, with learning framed as finding a minimal set of irreducible atoms that satisfy training constraints.
• The Sparse Crossing operation is used to generate candidate algebraic solutions by combining atoms while enforcing trace and trace constraint rules.
• Reduction operations simplify representations by eliminating redundant atoms, ensuring minimal cardinality.
• Atomization is performed via a stochastic, discrete algorithm without floating-point operations, avoiding local minima issues of gradient-based methods.
• The algorithm uses batch training to iteratively refine atom sets, favoring solutions with fewer atoms.
• Symmetries in input data (e.g., row/column permutations in images) are reflected in the structure of learned atoms, enabling derivation of new atoms via symmetry operations.

Experimental results

Research questions

• RQ1 Can a parameter-free, discrete algebraic method achieve generalization without function minimization or overfitting?
• RQ2 What is the relationship between the size of the algebraic representation (number of atoms) and the resulting test error rate?
• RQ3 How do symmetries in the input data manifest in the structure of learned algebraic atoms?
• RQ4 Can algebraic learning generalize effectively even with noisy or miss-labeled training data?
• RQ5 To what extent can symmetry exploitation generate new valid atoms without additional training?

Key findings

• The error rate in test data is inversely proportional to the compression rate (number of atoms), with the relationship validated experimentally on even-odd vertical bar classification.
• For 7×7 and 10×10 grids, the measured proportionality constant between error and inverse compression was within 10% and 5% of the theoretical prediction after accounting for symmetry-induced degeneracy.
• Algebraic learning achieved ~99% accuracy in classifying handwritten digits with no overfitting, even with miss-labeled training data and no explicit accuracy objective.
• The number of generalizing representations is astronomically large, yet the algorithm only needs to find one, demonstrating robustness and scalability.
• Symmetries in the data (e.g., row/column permutations) lead to symmetric atoms, enabling 'free' generation of new atoms via symmetry operations, improving efficiency.
• Convergence to theoretical predictions occurs as error rates decrease, with Sparse Crossing algorithm approaching theoretical error-compression bounds at low error levels.