[Paper Review] Embedding-based Retrieval with LLM for Effective Agriculture Information Extracting from Unstructured Data

Ruoling Peng, Kang Liu+3 more|arXiv (Cornell University)|Aug 6, 2023

Smart Agriculture and AI22 citations

TL;DR

The paper proposes FINDER, a pipeline combining embedding-based retrieval and general LLMs to extract structured agricultural information from unstructured documents without domain-specific training, evaluated on pest data with zero-shot prompts.

ABSTRACT

Pest identification is a crucial aspect of pest control in agriculture. However, most farmers are not capable of accurately identifying pests in the field, and there is a limited number of structured data sources available for rapid querying. In this work, we explored using domain-agnostic general pre-trained large language model(LLM) to extract structured data from agricultural documents with minimal or no human intervention. We propose a methodology that involves text retrieval and filtering using embedding-based retrieval, followed by LLM question-answering to automatically extract entities and attributes from the documents, and transform them into structured data. In comparison to existing methods, our approach achieves consistently better accuracy in the benchmark while maintaining efficiency.

Motivation & Objective

• Reduce reliance on domain-specific labeled data for agricultural information extraction.
• Convert unstructured agricultural texts into structured JSON entities and attributes.
• Evaluate a pipeline combining embedding-based retrieval with LLM-based extraction and binding.
• Demonstrate zero-shot performance of a general LLM on agricultural IE tasks.

Proposed method

• Split documents into small segments for embedding-based retrieval in a vector database.
• Use kNN-based retrieval with a median-distance filtering strategy to select relevant text segments.
• Stage 1: Ingest text and have an LLM identify descriptive words describing entities.
• Stage 2: Convert descriptive words into attribute types via LLM prompting and aggregation to mitigate inconsistency.
• Stage 3: Use LLM to perform Named Entity Recognition to identify entities of interest (physical objects in context).
• Stage 4: Reuse EBR to gather sentences and have LLM bind entities to attributes, outputting JSON per entity.

Experimental results

Research questions

• RQ1 Can a general pre-trained LLM extract structured agricultural information from unstructured documents without domain-specific training?
• RQ2 Does embedding-based retrieval improve the relevance of text segments used by the LLM for information extraction?
• RQ3 What is the accuracy of the multi-stage LLM-based extraction pipeline under zero-shot conditions, and how does the notion of 'acceptable' answers affect reported metrics?

Key findings

• Zero-shot LLM-based extraction achieves high entity extraction accuracy in Stage 3.
• Allowing 'acceptable' answers increases precision and recall substantially (up to 13 percentage points in some cases).
• Stage 2 attributes and Stage 4 binding show strong performance when evaluated with human judgments and acceptable-answer criteria.
• The overall pipeline outputs JSON with entities, attributes, and descriptions suitable for structured data use in agriculture.
• The EXIT pipeline demonstrates consistent accuracy improvements over baseline expectations in the tested materials.