[论文解读] Stabilizing Iterative Self-Training with Verified Reasoning via Symbolic Recursive Self-Alignment

Xinyu Zhang|arXiv (Cornell University)|Mar 23, 2026

Topic Modeling被引用 0

一句话总结

NSRSA 为迭代自训练增加了符号化验证子系统，在推理步级别筛选训练数据以防止错误传播，从而实现更强的 GSM8K 性能和跨任务正向迁移。

ABSTRACT

Recursive self-improvement--where a model iteratively trains on its own outputs--promises sustained capability growth but faces a fundamental obstacle: recursive drift. As models train on self-generated data across multiple iterations, errors in intermediate reasoning compound, leading to mode collapse and performance degradation. We propose Neuro-Symbolic Recursive Self-Alignment (NSRSA), which stabilizes iterative self-training by embedding a symbolic verification subsystem that gates training data quality at the reasoning step level. Unlike outcome-only filtering (which admits "lucky guesses" with flawed reasoning), NSRSA verifies each arithmetic operation via sympy, checks logical flow consistency across reasoning steps, and enforces domain constraints. We evaluate NSRSA on GSM8K using Qwen3-4B-Thinking across 5 self-training iterations under five conditions: no verification, outcome verification, majority voting, full NSRSA symbolic verification, and NSRSA with DPO. Our filtering analysis shows that NSRSA rejects approximately 34% of correct-answer solutions that pass outcome verification, eliminating "lucky guesses" with flawed reasoning from the training set. We further demonstrate that constructing DPO preference pairs from NSRSA verification teaches the model to distinguish sound from flawed reasoning (reward accuracy 46% to 63%). NSRSA provides an extensible framework that demonstrates how external symbolic verification can make recursive self-improvement measurable and reliable within domains where automated verification is available.

研究动机与目标

• 在自生成数据中推动递归自我改进，同时解决递归漂移问题。
• 引入一个逐步级别的符号化验证框架以门控训练数据质量。
• 证明符号化验证的推理能带来更稳定、可靠的迭代递归。
• 展示基于验证的学习如何提升跨任务迁移并提供可复现的管线。

提出的方法

• 将一个符号化验证子系统嵌入自训练循环，以通过四项检查门控训练数据：答案正确性、通过 sympy 的算术验证、逻辑流程一致性、以及领域约束满足性。
• 比较四种验证策略：无验证、仅结果验证、多数投票、以及完整的 NSRSA 符号化验证（可选 DPO）。
• 使用 NSRSA 在微调前筛选自生成的解，然后在 GSM8K 的 Qwen3-4B-Thinking 上进行 5 次自训练迭代评估。
• 从 NSRSA 验证通过的解与验证失败的解中构造 Direct Preference Optimization (DPO) 对来教学合理推理而非侥幸猜测。
• 提供可复现的管线，包括数据生成、验证、训练和评估。

Figure 1: NSRSA pipeline. At each iteration, the model generates multiple solutions per problem. The symbolic verification subsystem checks answer correctness, arithmetic validity (via sympy ), logical flow consistency, and domain constraints. Only solutions passing all checks enter the training set

实验结果

研究问题

• RQ1 逐步层面的符号化验证是否相较仅结果验证在迭代自训练中减少递归漂移？
• RQ2 NSRSA 如何影响 GSM8K 的精度、自洽性和模式多样性在多次自训练迭代中的表现？
• RQ3 符号化验证的推理能否改善对 MATH-500 的跨任务迁移，并从 DPO 偏好学习中受益？

主要发现

• NSRSA 在 5 次迭代中保持准确性增长，在 GSM8K 达到 91.0%，而无验证出现崩溃、仅结果验证趋于平台期。
• NSRSA 约拒绝了通过结果验证的约 34% 的正确答案解，从训练数据中去除了有缺陷的推理。
• 基于 NSRSA 派生偏好进行训练的 DPO 将奖励准确率从 46% 提升至 63%，并带来 91.2% 的 GSM8K 准确率（相比 NSRSA 无 DPO 的 91.0%）。
• NSRSA 实现对 MATH-500 的正向跨任务迁移，准确性从 45.5% 提升至 51.2%（+5.7 个百分点）。
• NSRSA 在迭代中保持解的多样性（较低的 Self-BLEU），相比仅结果验证方法表现出更少的模式崩溃。

Figure 2: GSM8K accuracy across 5 self-training iterations. NSRSA (green) enables stable recursive improvement. Outcome verification (orange) plateaus after iteration 2. No verification (red) collapses by iteration 3.