[Paper Review] Environment Maps: Structured Environmental Representations for Long-Horizon Agents

Yenchia Feng, Chirag Sharma+1 more|arXiv (Cornell University)|Mar 24, 2026

Scientific Computing and Data Management0 citations

TL;DR

The paper introduces Environment Maps, a persistent, agent-agnostic knowledge base that organizes contexts, parameterized actions, workflows, and tacit knowledge from multimodal traces into a queryable graph, and shows it substantially improves long-horizon web automation across multiple domains.

ABSTRACT

Although large language models (LLMs) have advanced rapidly, robust automation of complex software workflows remains an open problem. In long-horizon settings, agents frequently suffer from cascading errors and environmental stochasticity; a single misstep in a dynamic interface can lead to task failure, resulting in hallucinations or trial-and-error. This paper introduces $ extit{Environment Maps}$: a persistent, agent-agnostic representation that mitigates these failures by consolidating heterogeneous evidence, such as screen recordings and execution traces, into a structured graph. The representation consists of four core components: (1) Contexts (abstracted locations), (2) Actions (parameterized affordances), (3) Workflows (observed trajectories), and (4) Tacit Knowledge (domain definitions and reusable procedures). We evaluate this framework on the WebArena benchmark across five domains. Agents equipped with environment maps achieve a 28.2% success rate, nearly doubling the performance of baselines limited to session-bound context (14.2%) and outperforming agents that have access to the raw trajectory data used to generate the environment maps (23.3%). By providing a structured interface between the model and the environment, Environment Maps establish a persistent foundation for long-horizon planning that is human-interpretable, editable, and incrementally refinable.

Motivation & Objective

• Motivate and address cascading errors and environmental drift in long-horizon web automation.
• Propose a unified, persistent representation that combines diverse evidence sources into a graph.
• Demonstrate the benefits of environment maps for long-horizon planning across multiple domains.
• Show that environment maps outperform baselines and raw trajectory usage under a fixed agent stack.
• Discuss interpretability, maintenance, and extensibility advantages of the map-based approach.

Proposed method

• Define an environment map as a 4-tuple M = (C, A, W, K) with contexts, actions, workflows, and tacit knowledge.
• Develop a five-phase pipeline to construct maps from raw observations and interface representations: unify step sequences, extract actions, generalize actions to parameterized forms, extract contexts and tacit knowledge, and merge into a stable map.
• Cluster contexts by URL patterns, normalize URLs, and ground actions to parameterized templates with parameter values.
• Use an agent-agnostic evaluation on WebArena with three conditions (Baseline, Trajectory Access, Environment Map Access) to measure impact of maps.
• Provide JSON schemas and detailed appendix visuals to illustrate map structure and provenance.

Experimental results

Research questions

• RQ1 Does providing environment maps improve long-horizon agent success compared to baselines and raw trajectories?
• RQ2 How does an environment map affect generalization to tasks not directly demonstrated?
• RQ3 What is the impact of environment maps across UI-dense versus simpler web environments?
• RQ4 Are maps portable across sessions and maintenance-friendly in drifting UI environments?

Key findings

• Environment maps yield a 28.2% task success rate, nearly doubling the baseline 14.2%.
• Environment maps outperform using raw trajectories (23.3%) by about 4.9 percentage points.
• Maps improve targeted knowledge retrieval, with most tool calls being file-related (Read, Grep, Glob).
• Performance gains are largest in high-branching, UI-dense environments (e.g., GitLab, CMS).
• Maps remain interpretable, editable, and incrementally refinable, and can be constructed at low cost with reusable benefits across models.