[论文解读] The Elements of Differentiable Programming

Mathieu Blondel, Vincent Roulet|arXiv (Cornell University)|Mar 21, 2024

Scheduling and Optimization Algorithms被引用 7

一句话总结

本文综述了可微分编程，主张对程序进行端到端微分可以实现基于梯度的优化和通过概率执行进行不确定性量化，并将优化与概率视角联系起来。

ABSTRACT

Artificial intelligence has recently experienced remarkable advances, fueled by large models, vast datasets, accelerated hardware, and, last but not least, the transformative power of differentiable programming. This new programming paradigm enables end-to-end differentiation of complex computer programs (including those with control flows and data structures), making gradient-based optimization of program parameters possible. As an emerging paradigm, differentiable programming builds upon several areas of computer science and applied mathematics, including automatic differentiation, graphical models, optimization and statistics. This book presents a comprehensive review of the fundamental concepts useful for differentiable programming. We adopt two main perspectives, that of optimization and that of probability, with clear analogies between the two. Differentiable programming is not merely the differentiation of programs, but also the thoughtful design of programs intended for differentiation. By making programs differentiable, we inherently introduce probability distributions over their execution, providing a means to quantify the uncertainty associated with program outputs.

研究动机与目标

• 在AI进展背景下，推动可微分编程的必要性。
• 全面综述对可微分编程有用的基本概念。
• 采用并阐明两大视角：优化与概率，并在二者之间给出清晰的类比。
• 论证可微分编程需要经过深思熟虑的程序设计，以实现微分并量化不确定性。

提出的方法

• 对基础概念的综述：自动微分、图模型、优化与统计。
• 阐述两种互补视角（优化和概率）及其相应的类比。
• 讨论可微分执行如何在输出上产生概率分布以量化不确定性。
• 强调面向梯度方法、可微分且与之相容的程序设计原则。

实验结果

研究问题

• RQ1 哪些是使可微分编程成为可能的核心概念与工具？
• RQ2 如何在可微分编程中整合优化与概率视角？
• RQ3 将程序变为可微分如何实现对其输出不确定性的量化？
• RQ4 哪些原则引导旨在可微分的程序设计？

主要发现

• 可微分编程使复杂程序的端到端微分成为可能，包括具有控制流和数据结构的程序。
• 该领域以自动微分、图模型、优化和统计学为基础。
• 提出两种视角——优化与概率——并给出它们之间的清晰类比。
• 使程序可微分本质上会在执行过程中引入概率分布，从而实现不确定性量化。
• 本工作强调，可微分编程不仅关于微分，还关于为微分而进行的深思熟虑的程序设计。