[Paper Review] Informed Pre-Training on Prior Knowledge

Laura von Rueden, Sebastian Houben+3 more|arXiv (Cornell University)|May 23, 2022

Topic Modeling4 citations

TL;DR

This paper introduces informed pre-training on prior knowledge by transforming formal knowledge representations—such as graphs, equations, or templates—into compact knowledge prototypes for neural network pre-training. The method accelerates convergence, boosts generalization by up to 11% on small-data image classification, and enhances out-of-distribution robustness by 15%, with improvements primarily driven by high-level semantic features in deeper network layers, demonstrating a novel form of semantic transfer distinct from traditional data-based pre-training.

ABSTRACT

When training data is scarce, the incorporation of additional prior knowledge can assist the learning process. While it is common to initialize neural networks with weights that have been pre-trained on other large data sets, pre-training on more concise forms of knowledge has rather been overlooked. In this paper, we propose a novel informed machine learning approach and suggest to pre-train on prior knowledge. Formal knowledge representations, e.g. graphs or equations, are first transformed into a small and condensed data set of knowledge prototypes. We show that informed pre-training on such knowledge prototypes (i) speeds up the learning processes, (ii) improves generalization capabilities in the regime where not enough training data is available, and (iii) increases model robustness. Analyzing which parts of the model are affected most by the prototypes reveals that improvements come from deeper layers that typically represent high-level features. This confirms that informed pre-training can indeed transfer semantic knowledge. This is a novel effect, which shows that knowledge-based pre-training has additional and complementary strengths to existing approaches.

Motivation & Objective

• To address the challenge of training deep neural networks with limited labeled data by incorporating structured prior knowledge.
• To explore whether pre-training on concise, formal knowledge representations can improve model performance beyond standard data-based pre-training.
• To investigate whether knowledge-based pre-training transfers semantic, high-level features rather than low-level patterns.
• To evaluate the complementary benefits of combining data-based (e.g., ImageNet) and knowledge-based pre-training.
• To develop a universal, domain-agnostic method for initializing models using formal knowledge as prototypes.

Proposed method

• Transform formal prior knowledge—such as graphs, equations, or visual templates—into a small, condensed dataset of knowledge prototypes.
• Pre-train a neural network on these knowledge prototypes before fine-tuning on real, scarce training data.
• Use a standard deep learning architecture (e.g., CNN) and apply transfer learning by analyzing layer-wise contributions to performance.
• Compare the impact of pre-training on knowledge prototypes against baseline methods, including random initialization, ImageNet pre-training, and concurrent knowledge injection.
• Apply the method to both classification (GTSRB, MNIST, USPS) and regression (NOAA CO2) tasks to assess generalizability.
• Analyze feature transfer by measuring test accuracy when progressively freezing and transferring layers from the pre-trained model.

Experimental results

Research questions

• RQ1 Can pre-training on formal, concise knowledge prototypes improve model performance in low-data regimes?
• RQ2 Does knowledge-based pre-training transfer semantic, high-level features rather than low-level visual patterns?
• RQ3 How does informed pre-training compare to traditional data-based pre-training in terms of performance and layer-level contributions?
• RQ4 Can knowledge-based pre-training be combined with data-based pre-training to achieve further improvements?
• RQ5 Is the proposed method applicable across diverse domains and knowledge representation types?

Key findings

• Pre-training on knowledge prototypes improved test accuracy by up to 11% on small-data image classification benchmarks such as MNIST and GTSRB.
• The method increased out-of-distribution robustness by 15%, indicating stronger generalization to unseen data distributions.
• Performance gains were primarily driven by deeper network layers, which represent high-level semantic features, confirming that semantic knowledge is effectively transferred.
• In contrast to standard pre-training, which benefits early layers, informed pre-training enhances late layers, revealing a novel mechanism of 'informed transfer learning'.
• Combining ImageNet pre-training with subsequent knowledge-based pre-training yielded an additional 13% increase in test accuracy, demonstrating complementary strengths.
• The method outperformed concurrent knowledge injection and was most effective when used as a pre-training step, preserving flexibility for adaptation to new data.