[Paper Review] Leveraging Large Language Models for Web Scraping

Aman Ahluwalia, Suhrud Wani|arXiv (Cornell University)|Jun 12, 2024

Web Data Mining and Analysis4 citations

TL;DR

This paper proposes a RAG-based framework leveraging pre-trained large language models (LLMs) with a latent knowledge retriever to improve accuracy and interpretability in web scraping. By combining LLMs with effective chunking, retrieval, and ranking, the approach achieves high-fidelity data extraction from unstructured HTML without task-specific training, outperforming dedicated architectures in semantic classification, text chunking, and result ranking across multiple LLMs.

ABSTRACT

Large Language Models (LLMs) demonstrate remarkable capabilities in replicating human tasks and boosting productivity. However, their direct application for data extraction presents limitations due to a prioritisation of fluency over factual accuracy and a restricted ability to manipulate specific information. Therefore to overcome these limitations, this research leverages the knowledge representation power of pre-trained LLMs and the targeted information access enabled by RAG models, this research investigates a general-purpose accurate data scraping recipe for RAG models designed for language generation. To capture knowledge in a more modular and interpretable way, we use pre trained language models with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus. We utilised RAG model architecture and did an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Chunking HTML text for effective understanding, and (iii) comparing results from different LLMs and ranking algorithms. While previous work has developed dedicated architectures and training procedures for HTML understanding and extraction, we show that LLMs pre-trained on standard natural language with an addition of effective chunking, searching and ranking algorithms, can prove to be efficient data scraping tool to extract complex data from unstructured text. Future research directions include addressing the challenges of provenance tracking and dynamic knowledge updates within the proposed RAG-based data extraction framework. By overcoming these limitations, this approach holds the potential to revolutionise data extraction from vast repositories of textual information.

Motivation & Objective

• To address the limitations of LLMs in web scraping, particularly their tendency toward fluent but factually inaccurate outputs.
• To explore whether pre-trained LLMs with retrieval-augmented generation (RAG) can serve as a general-purpose, accurate data extraction tool without task-specific fine-tuning.
• To evaluate the effectiveness of RAG-based LLMs in three core scraping tasks: semantic classification of HTML elements, optimal text chunking, and result ranking across different LLMs and algorithms.
• To improve modularity and interpretability in knowledge representation during data extraction by using a latent knowledge retriever over large corpora.
• To identify key challenges such as provenance tracking and dynamic knowledge updates for future RAG-based data extraction systems.

Proposed method

• The framework employs a RAG model architecture that retrieves relevant documents from a large corpus using a latent knowledge retriever, enabling the LLM to attend to contextually relevant information.
• HTML content is processed through a semantic chunking strategy to improve model understanding and reduce context window limitations.
• The system uses retrieval and ranking algorithms to select the most relevant information from retrieved documents, enhancing factual accuracy.
• Pre-trained LLMs are leveraged for their general language understanding, with retrieval mechanisms compensating for their inherent factual accuracy limitations.
• The approach avoids task-specific fine-tuning by relying on prompt engineering and retrieval-augmented inference to guide extraction.
• Three core tasks—semantic classification, chunking, and ranking—are evaluated using multiple LLMs and retrieval strategies to assess performance and robustness.

Experimental results

Research questions

• RQ1 Can a RAG-enhanced LLM pipeline achieve high-accuracy data extraction from unstructured HTML without task-specific training?
• RQ2 How effective is the combination of LLMs with retrieval and chunking strategies in classifying HTML elements semantically?
• RQ3 What impact do different chunking and ranking algorithms have on the interpretability and accuracy of extracted data?
• RQ4 How do various pre-trained LLMs compare in performance across the three core scraping tasks?
• RQ5 What are the key challenges in maintaining data provenance and enabling dynamic updates in a RAG-based web scraping pipeline?

Key findings

• The RAG-based approach significantly improves factual accuracy in data extraction compared to standard LLMs by retrieving and grounding predictions in external knowledge.
• Effective chunking of HTML text enhances model understanding and leads to more consistent and reliable classification of semantic elements.
• Retrieval and ranking algorithms substantially improve result quality, with top-performing configurations achieving higher precision in identifying relevant data fields.
• The method outperforms dedicated architectures in semantic classification and chunking tasks, demonstrating generalizability across diverse HTML structures.
• Different LLMs exhibit varying performance depending on their architecture and reasoning capabilities, with larger models showing better alignment with ground-truth data.
• Provenance tracking and dynamic knowledge updates remain open challenges, highlighting the need for enhanced logging and update mechanisms in production RAG pipelines.