[Paper Review] ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data

Yuanbo Tang, Huaze Tang+7 more|arXiv (Cornell University)|Feb 4, 2026

Personal Information Management and User Behavior0 citations

TL;DR

ProAgentBench introduces a real-world, privacy-preserving dataset and a two-stage framework to evaluate proactive AI agents on when to assist and how to assist, showing real-world data and long-term context improve performance.

ABSTRACT

Proactive agents that anticipate user intentions without explicit prompts represent a significant evolution in human-AI interaction, promising to reduce cognitive load and streamline workflows. However, existing datasets suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. To address these gaps, we introduce ProAgentBench, a rigorous benchmark for proactive agents in working scenarios. Our contributions include: (1) a hierarchical task framework that decomposes proactive assistance into timing prediction and assist content generation; (2) a privacy-compliant dataset with 28,000+ events from 500+ hours of real user sessions, preserving bursty interaction patterns (burstiness B=0.787) absent in synthetic data; and (3) extensive experiments that evaluates LLM- and VLM-based baselines. Numerically, we showed that long-term memory and historical context significantly enhance prediction accuracy, while real-world training data substantially outperforms synthetic alternatives. We release our dataset and code at https://anonymous.4open.science/r/ProAgentBench-6BC0.

Motivation & Objective

• Create a benchmark for proactive agents in real-workflow scenarios.
• Capture authentic, long-term user interaction data with pre-assistance context.
• Formulate a two-stage framework (When to Assist, How to Assist) for systematic evaluation.
• Quantify the impact of real-world data and long-term memory on proactive intervention.
• Provide baselines across LLMs and VLMs to guide future research.

Proposed method

• Develop a hierarchical “When + How” framework to decompose proactive assistance into timing prediction and content generation.
• Assemble a privacy-compliant dataset with 28,528 events from 500+ hours of real user sessions, preserving bursty interaction patterns.
• Use a data collection pipeline with anonymization, human-in-the-loop review, and automatic LLM-based event annotation.
• Evaluate a diverse set of LLM and VLM baselines, including prompt-based (Zero-shot, CoT, Self-Consistency) and memory-based (RAG, Knowledge Graph, Clustering) approaches.
• Analyze the impact of historical observation length and long-term user context on prediction and generation tasks.

Figure 1 : Illustration of Proactive Agent Workflow. The agent continuously monitors user screen activities and contextual signals. When assistance is needed, it proactively determines when to intervene and how to assist based on historical observations and user behavior patterns.

Experimental results

Research questions

• RQ1 How does historical observation sequence length affect When to Assist and How to Assist performance?
• RQ2 What is the impact of incorporating long-term user context (memory) on proactive aid, and which memory strategy is most effective?
• RQ3 Does real-world training data outperform synthetic data for fine-tuning proactive agents?
• RQ4 To what extent do prompting strategies (Zero-shot, CoT, Self-Consistency) help or hinder performance across models?
• RQ5 What are the practical metrics that reflect real-world productivity in proactive intervention?

Key findings

• Longer historical context improves timing and intention prediction, with diminishing returns after about 5 minutes.
• Knowledge Graph-based long-term memory yields the largest performance gains over zero-shot baselines (Accuracy +11.8%, Intention Accuracy +26.9%, F1 +6.1%).
• Real-world training data substantially outperforms synthetic data for fine-tuning across models.
• Prompting strategies have mixed effects; chain-of-thought can help larger models but harms smaller/open models, and self-consistency provides limited gains.
• Model performance on semantic similarity for How to Assist remains relatively low, indicating room for improvement in content generation quality.

(a) Weekday distribution.