[论文解读] Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior

Shashank Subramanian, Peter Harrington+5 more|arXiv (Cornell University)|Jun 1, 2023

Topic Modeling被引用 14

一句话总结

本文研究了用于 SciML 的神经操作符的预训练和微调，表明预训练模型在下游数据显著更少的情况下也能实现准确迁移，并且通过扩大模型规模可以提升在各种 PDE 任务上的微调收益。

ABSTRACT

Pre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV). Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre-trained model size is scaled, (ii) the downstream training dataset size is scaled, (iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that-when fine-tuned appropriately-transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behavior across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential of the "pre-train and fine-tune" paradigm for SciML problems, demonstrating a path towards building SciML foundation models. We open-source our code for reproducibility.

研究动机与目标

• 建立一个框架，用于研究在多样化 PDE 系统中使用神经算子进行 SciML 传输学习。
• 量化下游数据规模、模型尺寸和物理参数如何影响 TL（传输学习）性能。
• 评估多算子预训练以及在同域和跨域设置中的零-shot/少样本微调效果。

提出的方法

• 在大量、多样化的 PDE 问题上对 Fourier Neural Operator (FNO) 进行预训练，问题具有变化的源项和系数。
• 引入每实例归一化以处理输入数值范围广泛的问题。
• 在下游任务上使用零-shot 或少样本学习对预训练模型进行微调，并与从头训练进行比较。
• 通过调整嵌入维度和傅里叶模态来改变模型规模，以研究缩放效应。
• 通过将系数和源项作为输入，评估跨不同物理场景和跨多算子的迁移。

实验结果

研究问题

• RQ1 下游数据集规模对神经算子传输学习性能有何影响？
• RQ2 模型参数规模如何影响微调收益相对于从头训练的收益？
• RQ3 当下游物理过程与预训练物理过程不同（OOD泛化）时，迁移学习表现如何？
• RQ4 一个在多算子上进行预训练的单一模型能否有效迁移到跨不同算子的新的 PDE 任务？

主要发现

• 在多样化语料上进行预训练后再微调，性能显著优于从头训练，尤其是在下游数据有限的情况下。
• 随着模型容量增大，微调相对于从头训练获得的收益更大。
• 在中等程度的分布外漂移以及少样本设置下，迁移学习仍然具有优势；当漂移变大或下游数据充足时，收益减弱。
• 一个在多个算子上预训练的单一模型在下游任务中保持迁移收益，为 SciML 提供类似 foundation-model 的方法。