[论文解读] DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs

Shidong Cao, Hongzhan Lin+3 more|arXiv (Cornell University)|Jan 7, 2026

Constraint Satisfaction and Optimization被引用 0

一句话总结

DiffCoT 将多步链式推理重新表述为带滑动窗口的扩散式去噪过程，通过回溯纠错来降低自回归大语言模型的暴露偏差。

ABSTRACT

Chain-of-Thought (CoT) reasoning improves multi-step mathematical problem solving in large language models but remains vulnerable to exposure bias and error accumulation, as early mistakes propagate irreversibly through autoregressive decoding. In this work, we propose DiffCoT, a diffusion-styled CoT framework that reformulates CoT reasoning as an iterative denoising process. DiffCoT integrates diffusion principles at the reasoning-step level via a sliding-window mechanism, enabling unified generation and retrospective correction of intermediate steps while preserving token-level autoregression. To maintain causal consistency, we further introduce a causal diffusion noise schedule that respects the temporal structure of reasoning chains. Extensive experiments on three multi-step CoT reasoning benchmarks across diverse model backbones demonstrate that DiffCoT consistently outperforms existing CoT preference optimization methods, yielding improved robustness and error-correction capability in CoT reasoning.

研究动机与目标

• Motivate reducing exposure bias and error accumulation in autoregressive CoT reasoning.
• Reformulate CoT as a globally revisable trajectory through a diffusion-inspired denoising process.
• Enable joint generation and revision of intermediate steps while preserving token-level autoregression.
• Introduce a step-level diffusion-noising and sliding-window denoising mechanism with causal diffusion noise.
• Demonstrate robustness and improved error correction on multiple mathematical reasoning benchmarks.

提出的方法

• Define CoT as a sequence of steps and view forward training as a diffusion-style denoising problem.
• Construct step-level forward noising by reward-ranking multiple candidate responses per step, creating a low-noise to high-noise trajectory set.
• Apply a diffusion sliding window to progressively denoise past steps while generating the next step, enabling retrospective correction.
• Impose a causal diffusion noise schedule where later steps receive stronger noise to encode temporal dependencies.
• Optimize with Direct Preference Optimization (DPO) on mixed (win/lose) sequences constructed from denoised and noisy prefixes.
• Fine-tune existing autoregressive models (LLMs) to integrate diffusion-style reasoning with minimal architectural changes.]
• research_questions Non-empty translated answers will be converted below:
• research_questions: ["Diffusion 风格的去噪过程是否能够缓解 LLMs 链式推理中的暴露偏差？","滑动窗口扩散方法是否能够在保持自回归生成的同时实现对中间步骤的有效回顾性修正？","因果扩散噪声调度如何影响多步推理的一致性与正确性？","相较于 Step-DPO 和 Full-Step-DPO，在不同骨干模型和数学基准上 DiffCoT 的表现如何？"]
• key_findings: ["DiffCoT 在三个公开数学推理基准上，与多种骨干模型相比，始终优于现有的偏好优化方法。","消融实验显示扩散窗口大小与因果连通性之间存在权衡；窗口过小或过大都会降低性能。","因果扩散噪声调度至关重要；中断噪声进程会显著损害准确性。","DiffCoT 提高对被污染前缀的鲁棒性，并展现出比以往方法更强的纠错能力。"]
• table_headers: []
• table_rows: []

实验结果

研究问题

• RQ1 Diffusion 风格的去噪过程是否能够缓解 LLMs 链式推理中的暴露偏差？
• RQ2 滑动窗口扩散方法是否能够在保持自回归生成的同时实现对中间步骤的有效回顾性修正？
• RQ3 因果扩散噪声调度如何影响多步推理的一致性与正确性？
• RQ4 DiffCoT 相对于 Step-DPO 和 Full-Step-DPO 在不同骨干和数学基准下的表现如何？

主要发现

• DiffCoT 在三个公开的数学推理基准上，针对多种骨干模型，始终优于现有的偏好优化方法。
• 消融表明扩散窗口大小与因果连通性之间存在权衡；窗口过小或过大均会降低性能。
• 因果扩散噪声调度至关重要；打乱噪声进程会显著降低准确性。
• DiffCoT 提高对被污染前缀的鲁棒性，并显示出比以往方法更强的错误纠正能力。