[Paper Review] Informational Neurobayesian Approach to Neural Networks Training. Opportunities and Prospects

Artem V. Artemov, Evgeniy Veniaminovich Lutsenko+2 more|arXiv (Cornell University)|Oct 19, 2017

Neural Networks and Applications4 citations

TL;DR

This paper introduces the Informational Neurobayesian Approach (INA), a novel method for training neural networks by interpreting synaptic weights as measures of information quantity rather than arbitrary values. By leveraging information theory and Bayesian updating, INA reduces computational complexity to O(log n) and enables training of large-scale models—such as a 6.3 billion-parameter word recognition system—on standard CPUs in under 3 hours, achieving F1 scores above 0.97 on text reconstruction tasks with minimal data and no GPUs.

ABSTRACT

A study of the classification problem in context of information theory is presented in the paper. Current research in that field is focused on optimisation and bayesian approach. Although that gives satisfying results, they require a vast amount of data and computations to train on. Authors propose a new concept named Informational Neurobayesian Approach (INA), which allows to solve the same problems, but requires significantly less training data as well as computational power. Experiments were conducted to compare its performance with the traditional one and the results showed that capacity of the INA is quite promising.

Motivation & Objective

• To address the high data and computational demands of traditional deep learning and Bayesian neural network training.
• To develop a method that interprets neural network weights as information quantities rather than arbitrary parameters.
• To enable training of large-scale neural models on standard CPUs without GPUs, reducing resource requirements.
• To improve performance on low-data and one-shot learning scenarios, particularly in natural language and text recognition tasks.
• To facilitate the construction of hierarchical, multi-level neural networks through additive information-based weight composition.

Proposed method

• Reinterprets neural network weights as information quantities—specifically, the logarithmic ratio of posterior to prior probabilities of class outcomes, based on Harkevich's formula.
• Applies the System Theory of Information (STI) to quantify the informational value of features in predicting outcomes, enabling information-based weight assignment.
• Uses an EM-like algorithm with E-step and M-step to optimize information emergence, where E-step computes information content and M-step updates model parameters.
• Employs a modified Bayesian framework that regularizes weights using prior distributions, reducing overfitting and enabling learning from limited data.
• Introduces a logarithmic complexity learning algorithm (O(log n) for E-step, O(n log n) for M-step), significantly reducing computational cost compared to gradient descent (O(n²)).
• Enables independent training of information neuromodels, allowing modular construction of deep, multi-level neural networks where each neuron's input is a lower-order information model.

Experimental results

Research questions

• RQ1 Can neural network training be redefined using information-theoretic principles to reduce data and computational requirements?
• RQ2 How does interpreting weights as information quantities improve model performance on low-data or one-shot learning tasks?
• RQ3 Can large-scale neural networks be effectively trained on standard CPUs without GPUs using an information-based optimization framework?
• RQ4 What is the impact of additive information properties on the composition of multi-layer neural networks?
• RQ5 How does the proposed method compare to classical gradient descent and neurobayesian approaches in terms of accuracy, speed, and resource efficiency?

Key findings

• The INA method achieved an F1 score of 0.98 on a Russian word recognition task using only 100,000 words, demonstrating strong performance on low-data, complex text tasks.
• A 6.3 billion-parameter neural model for Russian word recognition was trained in 2 hours and 35 minutes on a standard CPU-only server (Intel Xeon E5-1650 v3, 128 GB RAM), without using GPUs.
• The model achieved a perplexity of 1.26 per letter and bigram, indicating high accuracy in sequence modeling and information prediction.
• On MNIST, the model reached an F1 score of 0.81 in one epoch, significantly outperforming baseline results under similar constraints.
• The House Prices regression task achieved an RMSE of 0.42 in one epoch, compared to the best-known result of 0.06628, indicating strong potential despite not being optimized for peak performance.
• The INA algorithm's computational complexity is O(log n) for the E-step and O(n log n) for the M-step, representing a substantial improvement over classical gradient descent (O(n²)) and enabling efficient training on conventional hardware.