[论文解读] Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models

Hui Wu, Hengyi Cai+7 more|arXiv (Cornell University)|Feb 1, 2026

Explainable Artificial Intelligence (XAI)被引用 0

一句话总结

SAGE 在训练过程中动态选择并给偏好对进行打分，以最大化梯度效率与稳定性，在长链推理任务上超越静态 DPO 基线。

ABSTRACT

Preference-based alignment is pivotal for training large reasoning models; however, standard methods like Direct Preference Optimization (DPO) typically treat all preference pairs uniformly, overlooking the evolving utility of training instances. This static approach often leads to inefficient or unstable optimization, as it wastes computation on trivial pairs with negligible gradients and suffers from noise induced by samples near uncertain decision boundaries. Facing these challenges, we propose SAGE (Stability-Aware Gradient Efficiency), a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates. Concretely, SAGE integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence with a fine-grained, stability-aware scoring function that prioritizes informative, confident errors while filtering out unstable samples. Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines, highlighting the critical role of policy-aware, stability-conscious data selection in reasoning alignment.

研究动机与目标

• 推动在偏好基对齐中为长链推理模型引入动态、策略感知的数据选择的必要性。
• 引入 SAGE，一个两层框架，将粗粒度的池刷新与粒度更细的稳定性感知打分耦合。
• 证明稳定性意识、梯度高效的数据选择能够改善收敛性与数学推理基准的性能。
• 展示 SAGE 在模型规模上的数据效率和稳定优化。

提出的方法

• 引入一个粗粒度的可刷新池策略以创建随时间更新的在策略候选池。
• 开发一个细粒度的 SAGE 打分，通过牛顿启发、曲率感知代理来平衡梯度信号与预测置信度。
• 将 SAGE 目标定义为基于分数的硬过滤，并使用截断的高信噪比子集进行训练。
• 使用难度安排来影响池的组成（简单、中等、困难）并用长度归一化分数控制响应长度。
• 提供消融实验以展示池构建、信息性、梯度信号与曲率正则化的贡献。

Figure 1: Schematic illustration of dynamic sample utility in the stability-informativeness space. SAGE prioritizes samples with high signal quality and stable optimization behavior.

实验结果

研究问题

• RQ1 动态、策略自适应的数据选择是否能在推理模型对齐中提升稳定性与效率？
• RQ2 稳定性意识的效用分数是否比静态 DPO 提高梯度质量与收敛性？
• RQ3 粗粒度课程与细粒度 SAGE 打分对长就绪（long-CoT）数学推理基准有何影响？
• RQ4 模型规模与数据预算如何与 SAGE 的数据选择策略相互作用？

主要发现

Method	GSM8K	MATH500	Minerva	Gaokao	Olympiad	CollegeMath	AMC23	AIME24	Avg
Vanilla (Qwen2.5-1.5B-Instruct)	73.70	54.60	16.90	46.20	22.70	38.40	6.70	25.00	35.53
w/ DPO (Full)	74.70	56.20	19.50	47.30	20.00	38.00	10.00	22.50	36.03
w/ DPO (Random)	73.50	56.40	19.10	48.60	19.60	37.90	3.30	25.00	35.43
SAGE (Ours)	74.80	57.20	20.20	50.40	21.50	38.10	10.00	27.50	37.46
Vanilla (Qwen2.5-3B-Instruct)	86.90	65.20	25.70	56.40	27.70	44.50	6.70	47.50	45.08
w/ DPO (Full)	86.40	65.60	27.20	56.90	27.00	44.90	10.00	50.00	46.00
w/ DPO (Random)	87.00	65.20	26.10	56.40	26.50	45.00	0.00	45.00	43.90
SAGE (Ours)	87.50	66.00	28.30	58.23	27.70	45.14	13.30	55.00	47.65
Vanilla (Qwen2.5-7B-Instruct)	92.30	81.60	28.30	69.90	45.30	42.40	23.30	57.50	55.08
w/ DPO (Full)	92.70	82.00	29.40	70.60	46.50	42.70	26.70	62.50	56.64
w/ DPO (Random)	91.30	79.40	26.80	71.40	43.00	42.70	20.00	57.50	54.01
SAGE (Ours)	93.10	82.80	33.10	71.40	45.50	43.10	33.30	70.00	59.04

• SAGE 在 1.5B、3B、7B Qwen 模型上的八个数学推理基准中持续优于标准 DPO。
• 消融表明曲率正则化与梯度信号组件对增益至关重要，尤其在更难的基准上。
• SAGE 使梯度方差下降、优化轨迹更稳定，同时实现更高的最终准确率，优于 DPO。
• 中等保持比（gamma 在 [0.4,0.6]）在降低有效令牌数的同时获得更好准确度，兼顾计算与性能。
• SAGE 的增益在中等模型尺寸（3B）与更难任务上最为显著，表明对偏好监督的利用有所提升。

Figure 2: Overall framework of SAGE.