[Paper Review] OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents

Yuhang Zhou, Kai Zheng+5 more|arXiv (Cornell University)|Jan 26, 2026

Machine Learning in Materials Science0 citations

TL;DR

OffSeeker demonstrates that fully offline training (SFT + DPO) can match or rival online-RL systems for deep research agents, aided by the open-source DeepForge data suite and 8B-parameter models.

ABSTRACT

Deep research agents have shown remarkable potential in handling long-horizon tasks. However, state-of-the-art performance typically relies on online reinforcement learning (RL), which is financially expensive due to extensive API calls. While offline training offers a more efficient alternative, its progress is hindered by the scarcity of high-quality research trajectories. In this paper, we demonstrate that expensive online reinforcement learning is not all you need to build powerful research agents. To bridge this gap, we introduce a fully open-source suite designed for effective offline training. Our core contributions include DeepForge, a ready-to-use task synthesis framework that generates large-scale research queries without heavy preprocessing; and a curated collection of 66k QA pairs, 33k SFT trajectories, and 21k DPO pairs. Leveraging these resources, we train OffSeeker (8B), a model developed entirely offline. Extensive evaluations across six benchmarks show that OffSeeker not only leads among similar-sized agents but also remains competitive with 30B-parameter systems trained via heavy online RL.

Motivation & Objective

• Address the data and cost barriers in deep research agent development by providing an open synthesis framework (DeepForge).
• Release a large-scale, open dataset to support offline training (QA pairs, SFT trajectories, DPO pairs).
• Demonstrate that an offline-trained 8B-parameter model (OffSeeker) can achieve competitive performance on deep research benchmarks.
• Quantify online RL costs and show offline training can achieve strong results with lower costs and higher reproducibility.

Proposed method

• Introduce DeepForge as an end-to-end, lightweight pipeline for synthesizing large-scale deep research tasks via real-world web interaction.
• Create DeepForge-QA to yield 66k multi-step deep search questions with ground-truth answers.
• Collect 33k SFT trajectories and 21k DPO preference pairs for offline training.
• Train OffSeeker (8B) with fully offline supervised fine-tuning followed by offline Direct Preference Optimization (DPO).
• Evaluate on six deep research benchmarks (GAIA, BrowseComp, HLE, XBench-DeepSearch, WebWalkerQA) using pass@1 with LLM judges.
• Analyze impact of context window size, model scaling, and data quality on offline performance.

Figure 1 : Overview of our DeepForge data synthesis pipeline. DeepForge comprises two main stages: (a) Scalable Entity Expansion and (b) Complex Question Generation. After synthesizing complex deep search tasks, we further deploy an agent framework to collect high-quality trajectories.

Experimental results

Research questions

• RQ1 Can offline training (SFT + DPO) match or exceed online RL performance for deep research agents on standard benchmarks?
• RQ2 How does data synthesis quality (via DeepForge) influence offline learning outcomes?
• RQ3 What is the impact of context window size on deep-research tasks requiring long-horizon reasoning?
• RQ4 Do smaller offline models trained with high-quality offline data generalize to broader deep-research tasks?

Key findings

• OffSeeker achieves top-tier performance among agents of comparable scale and rivals 30B-parameter systems trained with online RL.
• Offline training with SFT + DPO yields consistent gains across benchmarks (GAIA, BrowseComp-zh, BrowseComp-en, HLE, XBench-DeepSearch, WebWalkerQA).
• Training costs are near zero with offline methods, avoiding API costs and rate-limit risks associated with online RL.
• OffSeeker-8B (DPO) reaches 26.6 on BrowseComp-zh and 61.7 on WebWalkerQA, approaching larger online-RL systems.
• Data quality from DeepForge enables efficient learning: larger SFT datasets improve accuracy, and offline data scales with training.
• Context window size significantly influences performance, with larger contexts delivering substantial gains on complex multi-hop tasks.

Figure 2 : Comparison of tool calling turns distribution between DeepForge -generated tasks and BrowseComp-en with DeepSeek-v3.1.