[Paper Review] DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing

Shreya Shankar, Chambers, Tristan+3 more|arXiv (Cornell University)|Oct 16, 2024

Semantic Web and Ontologies4 citations

TL;DR

DocETL is a declarative system that uses agent-based query rewriting and evaluation to improve the accuracy of complex document processing pipelines powered by large language models (LLMs). By applying novel rewrite directives, agent-guided plan evaluation, and an optimization algorithm, DocETL achieves 1.34x to 4.6x higher output quality than hand-engineered baselines on unstructured document tasks.

ABSTRACT

Analyzing unstructured data has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for declarative frameworks for LLM-powered processing of unstructured data. However, these frameworks focus on reducing cost when executing user-specified operations using LLMs, rather than improving accuracy, executing most operations as-is (in a single LLM call). This is problematic for complex tasks and data, where LLM outputs for user-defined operations are often inaccurate, even with optimized prompts. For example, an LLM may struggle to identify {\em all} instances of specific clauses, like force majeure or indemnification, in lengthy legal documents, requiring decomposition of the data, the task, or both. We present DocETL, a system that optimizes complex document processing pipelines, while accounting for LLM shortcomings. DocETL offers a declarative interface for users to define such pipelines and uses an agent-based approach to automatically optimize them, leveraging novel agent-based rewrites (that we call rewrite directives), as well as an optimization and evaluation framework. We introduce (i) logical rewriting of pipelines, tailored for LLM-based tasks, (ii) an agent-guided plan evaluation mechanism that synthesizes and orchestrates task-specific validation prompts, and (iii) an optimization algorithm that efficiently finds promising plans, considering the latencies of agent-based plan generation and evaluation. Our evaluation on four different unstructured document analysis tasks demonstrates that DocETL finds plans with outputs that are 25 to 80% more accurate than well-engineered baselines, addressing a critical gap in unstructured data analysis. DocETL is open-source at docetl.org, and as of March 2025, has amassed over 1.7k GitHub Stars, with users spanning a variety of domains.

Motivation & Objective

• Address the critical gap in existing LLM-powered declarative frameworks that prioritize cost reduction over accuracy in complex document processing.
• Overcome LLM limitations in handling long, complex documents—such as missing clauses or hallucinating content—by decomposing monolithic operations into more accurate, modular pipelines.
• Develop a system that autonomously optimizes document processing pipelines using LLM agents, rather than relying on static, user-provided prompts.
• Enable high-quality, comprehensive extraction and summarization of information from heterogeneous, lengthy documents like legal or police records, where accuracy is paramount.
• Provide a flexible, open-source framework that supports diverse unstructured data workloads beyond templated documents, extending the reach of LLM-based ETL pipelines.

Proposed method

• Introduces novel rewrite directives to logically decompose complex, error-prone LLM operations into simpler, more accurate sub-operations tailored for document processing.
• Employs an agent-based framework that synthesizes and evaluates task-specific validation prompts to assess the quality of rewritten pipeline plans.
• Uses an optimization algorithm that explores promising pipeline plans under time constraints, leveraging opportunistic rewrites during plan generation.
• Applies a verifier agent to determine whether a pipeline plan is sufficiently optimized, triggering further rewriting if needed.
• Supports a declarative YAML-based interface for users to specify document processing pipelines, abstracting low-level implementation details.
• Orchestrates a feedback loop where newly synthesized operations are immediately subject to further rewriting and evaluation, enabling iterative refinement.

Figure 1. Optimization for a pipeline designed to accomplish the task in Example 1.1 . The diagram illustrates the system mid-optimization of the initial map operation. DocETL employs LLMs to synthesize new plans using novel rewrite directives. The process begins with an LLM verifier determining if

Experimental results

Research questions

• RQ1 How can LLM-based document processing pipelines be systematically rewritten to improve accuracy on complex, long-form documents?
• RQ2 What types of logical rewrite directives are most effective for decomposing monolithic LLM operations into accurate, modular components?
• RQ3 Can agent-guided plan evaluation with task-specific validation prompts significantly improve the quality of LLM outputs in document analysis tasks?
• RQ4 To what extent can an optimization algorithm that balances plan quality and generation time outperform hand-crafted, baseline pipelines in real-world document processing?
• RQ5 How does the agent-driven, iterative rewriting process compare to static prompt engineering in terms of output comprehensiveness and accuracy?

Key findings

• DocETL achieves 1.34x to 4.6x higher output quality—measured in accuracy and comprehensiveness—than well-engineered baselines across three distinct unstructured document analysis tasks.
• The system’s agent-guided plan evaluation mechanism successfully identifies and mitigates common LLM failures such as missing clauses or hallucinated content in lengthy documents.
• Rewrite directives enable effective decomposition of complex operations, reducing the likelihood of LLM context overload and improving holistic understanding.
• The optimization algorithm efficiently explores the plan space within practical time limits, finding high-quality pipelines without requiring exhaustive search.
• The framework demonstrates robustness across diverse domains, including legal, medical, and social science document analysis, as validated by real-world use cases.
• As of October 2024, DocETL has gained over 800 GitHub stars, indicating strong adoption and utility across varied application domains.

Figure 2. Iterative folding in a reduce operation, illustrating the step-by-step evolution of both the scratchpad and accumulated output when processing multiple batches of documents. New input is integrated with the previous scratchpad state, updating mention counts and the intermediate output for