[Paper Review] A General Theory for Training Learning Machine

Hong Zhao|arXiv (Cornell University)|Apr 23, 2017

Neural Networks and Applications2 references3 citations

TL;DR

This paper proposes a general theoretical framework for training learning machines by systematically classifying prior knowledge into common and problem-dependent components, and introducing a design risk minimization principle to maximize their incorporation. It establishes a Monte Carlo-based algorithm to control input-output sensitivity through neuron-level adjustments, enabling robust training across function approximation, classification, and transductive inference tasks with clear guidelines for transfer functions, cost functions, and data preprocessing.

ABSTRACT

Though the deep learning is pushing the machine learning to a new stage, basic theories of machine learning are still limited. The principle of learning, the role of the a prior knowledge, the role of neuron bias, and the basis for choosing neural transfer function and cost function, etc., are still far from clear. In this paper, we present a general theoretical framework for machine learning. We classify the prior knowledge into common and problem-dependent parts, and consider that the aim of learning is to maximally incorporate them. The principle we suggested for maximizing the former is the design risk minimization principle, while the neural transfer function, the cost function, as well as pretreatment of samples, are endowed with the role for maximizing the latter. The role of the neuron bias is explained from a different angle. We develop a Monte Carlo algorithm to establish the input-output responses, and we control the input-output sensitivity of a learning machine by controlling that of individual neurons. Applications of function approaching and smoothing, pattern recognition and classification, are provided to illustrate how to train general learning machines based on our theory and algorithm. Our method may in addition induce new applications, such as the transductive inference.

Motivation & Objective

• To address the lack of foundational theory in machine learning, particularly regarding the role of prior knowledge, transfer functions, cost functions, and neuron biases.
• To develop a unified framework that maximizes incorporation of both common and problem-dependent prior knowledge during learning.
• To provide principled guidelines for selecting neural transfer functions, cost functions, and data preprocessing techniques.
• To control input-output sensitivity of learning machines by regulating individual neuron behavior.
• To enable new applications such as transductive inference through the proposed theoretical and algorithmic framework.

Proposed method

• Classify prior knowledge into common (general) and problem-dependent (specific) components.
• Propose a design risk minimization principle as the core learning objective to maximize prior knowledge utilization.
• Use a Monte Carlo algorithm to model and control the input-output response of learning machines.
• Control sensitivity of the overall system by adjusting the sensitivity of individual neurons.
• Assign roles to transfer functions, cost functions, and data preprocessing to enhance problem-dependent knowledge integration.
• Formulate neuron bias as a mechanism to adjust the effective input-output behavior of neurons, distinct from traditional interpretations.

Experimental results

Research questions

• RQ1 How can prior knowledge be systematically categorized and maximally incorporated into learning machines?
• RQ2 What is the theoretical basis for selecting neural transfer functions and cost functions in a principled manner?
• RQ3 How can input-output sensitivity of a learning machine be controlled through neuron-level design?
• RQ4 What role does neuron bias play in the learning process beyond standard activation functions?
• RQ5 Can the proposed framework support novel applications such as transductive inference?

Key findings

• The design risk minimization principle provides a theoretical foundation for maximizing the use of both common and problem-dependent prior knowledge in learning machines.
• The Monte Carlo algorithm enables accurate modeling of input-output responses, allowing systematic control of sensitivity across the network.
• Controlling individual neuron sensitivity effectively regulates the overall system's input-output behavior, improving robustness and generalization.
• The framework offers clear, principled guidelines for selecting transfer functions, cost functions, and data preprocessing steps based on prior knowledge.
• The theory supports new applications such as transductive inference, extending the scope of traditional learning machine paradigms.
• Empirical applications in function approximation, smoothing, and classification demonstrate the framework's effectiveness and versatility.