[论文解读] SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training

Kazem Meidani, Parshin Shojaee+2 more|arXiv (Cornell University)|Oct 3, 2023

Machine Learning in Materials ScienceMaterials Science被引用 3

一句话总结

SNIP 引入了一种统一的预训练框架，通过对比学习联合学习符号数学表达式及其数值对应形式，实现跨域表征对齐。通过利用共享的低维潜在空间表征，该方法在符号回归和数学性质预测任务中实现了最先进性能，尤其在少样本设置下表现优异，显著提升了准确率与模型简洁性。

ABSTRACT

In an era where symbolic mathematical equations are indispensable for modeling complex natural phenomena, scientific inquiry often involves collecting observations and translating them into mathematical expressions. Recently, deep learning has emerged as a powerful tool for extracting insights from data. However, existing models typically specialize in either numeric or symbolic domains, and are usually trained in a supervised manner tailored to specific tasks. This approach neglects the substantial benefits that could arise from a task-agnostic multi-modal understanding between symbolic equations and their numeric counterparts. To bridge the gap, we introduce SNIP, a Symbolic-Numeric Integrated Pre-training model, which employs contrastive learning between symbolic and numeric domains, enhancing their mutual similarities in the embeddings. By performing latent space analysis, we observe that SNIP provides cross-domain insights into the representations, revealing that symbolic supervision enhances the embeddings of numeric data and vice versa. We evaluate SNIP across diverse tasks, including symbolic-to-numeric mathematical property prediction and numeric-to-symbolic equation discovery, commonly known as symbolic regression. Results show that SNIP effectively transfers to various tasks, consistently outperforming fully supervised baselines and competing strongly with established task-specific methods, especially in the low data regime scenarios where available data is limited. Code and model are available at: https://github.com/deep-symbolic-mathematics/Multimodal-Math-Pretraining

研究动机与目标

• 通过统一的预训练实现符号数学与数值数据之间的相互理解，弥合两者之间的鸿沟。
• 开发一种与任务无关的模型，通过联合对比学习增强符号与数值领域中的表征能力。
• 改善符号回归与数学推理任务中的少样本与低数据场景。
• 通过预训练模型的可插值潜在空间支持高质量方程发现。

提出的方法

• SNIP 使用双分支基于 Transformer 的编码器分别处理符号表达式与数值数据。
• 采用联合对比学习目标对齐（符号，数值）对的嵌入表征，增强潜在空间中跨域的相似性。
• 模型在成对的符号与数值数据上进行预训练，无需任务特定的微调，从而支持零样本或少样本迁移。
• 预训练完成后，在数值编码器之上微调一个方程生成解码器，以执行符号回归。
• 采用潜在空间优化方法，通过导航连续的低维表征空间，寻找在准确率与复杂度之间达到平衡的方程。
• 该框架在合成数据集（Feynman）与真实世界数据集（Black-box）上进行评估，以检验泛化能力与鲁棒性。

Figure 1: The SNIP Framework: A schematic representation of the dual-encoder pre-training scheme for mutual learning between symbolic equations and their numerical observations. Both symbolic and numeric encoders work in tandem, capturing the paired similarities and essence of their respective modal

实验结果

研究问题

• RQ1 统一的预训练方法能否有效在共享潜在空间中对齐符号与数值表征？
• RQ2 符号与数值数据之间的联合对比学习是否能提升下游数学推理任务的性能？
• RQ3 与完全监督基线相比，SNIP 在少样本或低数据场景下的表现如何？
• RQ4 预训练的潜在空间在多大程度上支持通过潜在空间优化实现有效的符号回归？
• RQ5 与现有最先进符号回归方法相比，SNIP 是否能在准确率与模型复杂度之间实现更优平衡？

主要发现

• SNIP 在 SRBench 基准测试中达到最先进性能，在 Feynman 数据集上准确率排名前三，尤其在高准确率解（R² > 0.99）方面表现突出。
• 在 Black-box 数据集上，SNIP 的中位数 R² 得分为 0.872，位列第四，且平均复杂度得分最低（47.52），优于 Operon（64.95）、SBP-GP（639.19）与 E2E（82.78）。
• SNIP 在 Feynman 数据集上对不同噪声水平均保持稳定强性能，展现出对数据扰动的鲁棒性。
• 在帕累托最优性分析中，SNIP 在 Feynman 与 Black-box 数据集上均位于第一 Pareto 前沿，表明其在准确率与复杂度之间具有卓越的权衡性能。
• SNIP 超越了完全监督基线模型，并在少样本学习场景下与特定任务方法竞争激烈。
• 潜在空间分析表明，符号监督可提升数值表征质量，反之亦然，验证了通过联合预训练实现的相互增强机制。

Figure 2: 2D t-SNE representations of the encoded vectors across three model variants, colored for (top) Non-Convexity Ratio and (bottom) Function Upwardness prediction tasks.