[论文解读] A Universal Approximation Result for Difference of log-sum-exp Neural Networks

Giuseppe C. Calafiore, Stéphane Gaubert+2 more|arXiv (Cornell University)|May 21, 2019

Advanced Optimization Algorithms Research参考文献 33被引用 49

一句话总结

本文引入 Difference-LSE（DLSE）网络，证明它们对连续函数具有通用近似能力，并展示它们映射到无减法的 posynomial 表达式，便于 DC 型优化，同时给出一个基于数据驱动的糖尿病饮食设计示例。

ABSTRACT

We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node (LSE networks) is a smooth universal approximator of continuous functions over convex, compact sets. By using a logarithmic transform, this class of networks maps to a family of subtraction-free ratios of generalized posynomials, which we also show to be universal approximators of positive functions over log-convex, compact subsets of the positive orthant. The main advantage of Difference-LSE networks with respect to classical feedforward neural networks is that, after a standard training phase, they provide surrogate models for design that possess a specific difference-of-convex-functions form, which makes them optimizable via relatively efficient numerical methods. In particular, by adapting an existing difference-of-convex algorithm to these models, we obtain an algorithm for performing effective optimization-based design. We illustrate the proposed approach by applying it to data-driven design of a diet for a patient with type-2 diabetes.

研究动机与目标

• Motivate the need for surrogate models that are amenable to optimization in design tasks.
• Introduce Difference-LSE networks as a combination of two LSE networks to overcome convexity limitations.
• Prove universal approximation of continuous functions by DLSE_T and its rational-parameter variant.
• Show how a logarithmic transform maps DLSE_T to subtraction-free ratios of generalized posynomials.
• Discuss training and optimization strategies using DC programming and provide a data-driven design example.

提出的方法

• Define LSE and LSE_T functions and their convexity properties.
• Construct DLSE_T networks as the difference of two LSE_T networks.
• Prove universal approximation for DLSE_T over compact convex domains.
• Show mapping to subtraction-free ratios of generalized posynomials via a log transform.
• Demonstrate training via standard optimization methods and discuss DC programming applicability.

实验结果

研究问题

• RQ1 Can DLSE_T networks universally approximate any continuous function on a compact convex domain?
• RQ2 What is the role of the log-transform in relating DLSE_T outputs to subtraction-free posynomial expressions?
• RQ3 How can the DLSE_T structure facilitate optimization-based design through DC programming?
• RQ4 Can the approach be applied to real data-driven design problems, such as diet design for type-2 diabetes?

主要发现

• DLSE_T networks are universal smooth approximators of continuous functions on convex, compact sets.
• A log transform maps DLSE_T outputs to subtraction-free ratios of generalized posynomials, which also universal-approximate positive functions on log-convex compact sets.
• DLSE_T networks enable efficient optimization by exploiting a subtraction-free, DC-friendly structure.
• Rational-parameter LSE_T results extend universal approximation to rational schemes.
• A data-driven diet design example for type-2 diabetes illustrates the practical surrogate-design workflow.