[论文解读] Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation

Songen Gu, Yunuo Cai+3 more|arXiv (Cornell University)|Feb 11, 2026

Multimodal Machine Learning Applications被引用 0

一句话总结

论文提出 Dream4manip，这是一个三阶段框架，用于选择/改编高保真视频世界模型、将其蒸馏以实现快速少步推理，并使用就地条件化行动模型在上下文中实现长时序、指令驱动的机器人操作，从而提升实现能力、空间精度和任务成功率。

ABSTRACT

Robotic manipulation requires anticipating how the environment evolves in response to actions, yet most existing systems lack this predictive capability, often resulting in errors and inefficiency. While Vision-Language Models (VLMs) provide high-level guidance, they cannot explicitly forecast future states, and existing world models either predict only short horizons or produce spatially inconsistent frames. To address these challenges, we propose a framework for fast and predictive video-conditioned action. Our approach first selects and adapts a robust video generation model to ensure reliable future predictions, then applies adversarial distillation for fast, few-step video generation, and finally trains an action model that leverages both generated videos and real observations to correct spatial errors. Extensive experiments show that our method produces temporally coherent, spatially accurate video predictions that directly support precise manipulation, achieving significant improvements in embodiment consistency, spatial referring ability, and task completion over existing baselines. Codes & Models will be released.

研究动机与目标

• 推动鲁棒的预测世界模型用于长时序机器人操作，以预测环境演变。
• 将高保真视频预测与适合实时控制的快速推理相结合。
• 通过就地条件化行动模型，将想象的未来轨迹与真实观测对齐以纠正空间误差并生成可执行操作。

提出的方法

• 基于 embodiment 一致性和任务指标，选择 Cosmos-Predict2 作为世界模型骨干。
• 应用领域自适应与潜在空间对抗蒸馏，实现保留保真度的少步去噪。
• 通过将轨迹压缩为固定的一组关键帧并对视频扩散模型进行微调，引入长度无关的想象。
• 开发一个就地条件化行动模型，结合想象轨迹与真实观测来纠正空间误差并生成可执行操作。

Figure 1 : (a) The training pipeline for world model distillation. Additional details are provided in Section 3.1.2 . (b) The pipeline of our proposed policy model. Given the current observation and the instruction, the world model first generates imagined future frames. The in-context conditioned a

实验结果

研究问题

• RQ1 是否可以将高保真视频世界模型适应到机器人领域并蒸馏以实现快速推理，同时不牺牲时空保真度？
• RQ2 长度无关的想象是否能够实现对操控任务的可靠长时序预测？
• RQ3 就地条件化行动模型能否将想象的未来在真实观测中对齐，从而提升行动质量与任务成功率？
• RQ4 在将预测性世界模型与就地行动策略结合后，体现出在 embodiment 一致性、空间指向能力和任务完成方面的提升？

主要发现

• Dream4manip 在 LIBERO 基准的长时序操控上实现了最先进的性能。
• Dream4manip 在 LIBERO 上达到 98.2% 的总成功率，具备强泛化性：LIBERO-Spatial 99.4%、LIBERO-Object 99.2%、LIBERO-Goal 98.6%、LIBERO-Long 95.4%。
• 领域自适应加潜在对抗蒸馏提升了视频质量并实现高效少步推理（C-2B-DA+Dis: SSIM 0.84, PSNR 26.82, FVD 238.09）。
• 零-shot 空间指向能力在 Cosmos 模型中最强，领域自适应的 Cosmos 在各任务中实现了稳健的 embodiment 一致性。
• 在适应前，Cosmos-2B/14B 变体表现出更好的空间指向与操控能力，领域自适应版本则提升了时间一致性与保真度。

Figure 2 : The structure of the in-context conditioned action model. We use a transformer-based backbone inherited from ACT (Zhao et al. , 2023 ) for our action model, separated vision encoder is assembled to process videos and observations. The model will output an action chunk for each observation