[论文解读] Specializing Smaller Language Models towards Multi-Step Reasoning

Yao Fu, Hao Peng+3 more|arXiv (Cornell University)|Jan 30, 2023

Topic Modeling被引用 43

一句话总结

论文表明小模型（≤11B 参数）可以通过从大型教师模型蒸馏 CoT 数据进行专门化，在多步数学推理方面表现出色，权衡通用能力与目标任务性能，并在专门化后揭示对数线性缩放曲线。

ABSTRACT

The surprising ability of Large Language Models (LLMs) to perform well on complex reasoning with only few-shot chain-of-thought prompts is believed to emerge only in very large-scale models (100+ billion parameters). We show that such abilities can, in fact, be distilled down from GPT-3.5 ($\ge$ 175B) to T5 variants ($\le$ 11B). We propose model specialization, to specialize the model's ability towards a target task. The hypothesis is that large models (commonly viewed as larger than 100B) have strong modeling power, but are spread on a large spectrum of tasks. Small models (commonly viewed as smaller than 10B) have limited model capacity, but if we concentrate their capacity on a specific target task, the model can achieve a decent improved performance. We use multi-step math reasoning as our testbed because it is a very typical emergent ability. We show two important aspects of model abilities: (1). there exists a very complex balance/ tradeoff between language models' multi-dimensional abilities; (2). by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability. We further give comprehensive discussions about important design choices for better generalization, including the tuning data format, the start model checkpoint, and a new model selection method. We hope our practice and discoveries can serve as an important attempt towards specialized smaller models in the new research paradigm set by LLMs.

研究动机与目标

• 证明小型语言模型通过专门化能够实现强大的多步数学推理。
• 研究蒸馏和数据格式如何影响小模型的 CoT 能力。
• 描述通用能力（BBH）与目标特定（数学）能力之间的权衡。
• 在专门化后考察缩放行为和泛化性（分布内 vs 分布外）。
• 为有效的专门化小模型训练提供设计建议。

提出的方法

• 用由大型教师（code-davinci-002）生成的蒸馏数据对 FlanT5 和 T5 基线进行微调，以产生具备 CoT 的输出。
• 探索数据格式：上下文中仅答案、上下文中 CoT，以及零-shot 格式，以研究它们对能力的影响。
• 将分布匹配作为蒸馏目标以对齐学生和教师每步分布，通过动态规划解决分词器对齐问题。
• 采用基于对齐的动态规划方法对齐 GPT 和 T5 的标记化。
• 使用 GSM8K（分布内）和 4 个分布外数学数据集（MultiArith、ASDiv、SVAMP）以及 BigBench Hard 测试通用能力。
• 分析在不同微调阶段专门化进展与保留通用能力之间的权衡。

Figure 1: A. Model specialization process. Pretraining gives a strong base model (Raffel et al., 2020 ; Chowdhery et al., 2022 ) , instruction tuning elicits the model ability (Chung et al., 2022 ) , then specialization (this work’s focus) moves model abilities to a target direction. In this work, w

实验结果

研究问题

• RQ1 小模型（≤11B）通过专门化朝向 CoT 任务是否能实现增强的多步数学推理？
• RQ2 使用蒸馏数据格式和一个指令微调的基础模型对专门化性能有何影响？
• RQ3 专门化如何影响分布内与分布外的性能，以及零-shot 与上下文内能力？
• RQ4 在保留通用能力（BigBench Hard）与提升目标任务 CoT 数学推理之间会出现哪些权衡？
• RQ5 基于不同验证信号的模型选择如何影响在分布内和 OOD 任务上的最终表现？

主要发现

• 专门化通过专门化小模型的数学推理在 GSM8K 的平均准确率上提升约 +10，3B 和 11B 的 FlanT5 模型取得了不错的结果。
• 专门化的小模型在目标数学任务（GSM8K 和 OOD 数据集）上可以达到或接近更大模型的性能，但以牺牲 BigBench Hard 的通用能力为代价。
• 专门化小模型的缩放曲线变为对数线性（不是平坦的），表明在专门化后多步推理可以随模型规模平滑扩展。
• 指令微调的基线（FlanT5）在专门化后通常优于原始预训练基线（T5），强调从指令微调检查点出发的好处。
• 分布内与分布外的性能之间，以及上下文内与零-shot 能力之间存在明显的权衡，模型选择取决于所期望的泛化目标。
• 两种蒸馏策略在收敛速度上不同（分布匹配比采样匹配快），但最终性能差异不大。

Figure 2: X-axis means log of model scale, y-axis means validation accuracy on GSM8K. A : Previously, the community believe that small models has flat curve for both AO and CoT prompting and only when models become large enough the performance will have a “phase change” and suddenly increase. B : we