[Paper Review] Archiving Deferred Representations Using a Two-Tiered Crawling Approach

Justin F. Brunelle, Michele C. Weigle+1 more|arXiv (Cornell University)|Aug 10, 2015

Web Data Mining and Analysis5 references3 citations

TL;DR

This paper proposes a two-tiered web crawling strategy that uses a classifier to route deferred representations—those relying on client-side JavaScript—to PhantomJS for full rendering, while non-deferred resources are crawled with the faster Heritrix. The approach achieves a 5.2× speedup over pure PhantomJS crawling and expands the crawl frontier by 1.8× compared to Heritrix alone, significantly improving archival completeness without sacrificing performance.

ABSTRACT

Web resources are increasingly interactive, resulting in resources that are increasingly difficult to archive. The archival difficulty is based on the use of client-side technologies (e.g., JavaScript) to change the client-side state of a representation after it has initially loaded. We refer to these representations as deferred representations. We can better archive deferred representations using tools like headless browsing clients. We use 10,000 seed Universal Resource Identifiers (URIs) to explore the impact of including PhantomJS -- a headless browsing tool -- into the crawling process by comparing the performance of wget (the baseline), PhantomJS, and Heritrix. Heritrix crawled 2.065 URIs per second, 12.15 times faster than PhantomJS and 2.4 times faster than wget. However, PhantomJS discovered 531,484 URIs, 1.75 times more than Heritrix and 4.11 times more than wget. To take advantage of the performance benefits of Heritrix and the URI discovery of PhantomJS, we recommend a tiered crawling strategy in which a classifier predicts whether a representation will be deferred or not, and only resources with deferred representations are crawled with PhantomJS while resources without deferred representations are crawled with Heritrix. We show that this approach is 5.2 times faster than using only PhantomJS and creates a frontier (set of URIs to be crawled) 1.8 times larger than using only Heritrix.

Motivation & Objective

• To address the challenge of archiving deferred representations that rely on client-side JavaScript to render fully.
• To measure the performance trade-offs between traditional crawlers (Heritrix, wget) and JavaScript-executing crawlers (PhantomJS) in terms of crawl speed and frontier size.
• To develop and validate a tiered crawling strategy that minimizes performance overhead while maximizing discovery of embedded resources in deferred representations.
• To improve archival quality by ensuring complete memento creation through targeted use of headless browsing for JavaScript-dependent resources.

Proposed method

• A classifier is trained to predict whether a web resource contains deferred representations based on DOM features, achieving 79% accuracy.
• The two-tiered approach routes resources classified as deferred to PhantomJS for full client-side rendering and URI discovery, while non-deferred resources are crawled with Heritrix for speed.
• Crawling performance is evaluated using a dataset of 10,000 seed URIs, measuring crawl time, frontier size, and embedded resource discovery.
• PhantomJS is used to execute JavaScript and discover URIs that conventional crawlers miss, particularly embedded resources loaded dynamically.
• The frontier size and crawl time are compared across three crawlers: wget, Heritrix, and PhantomJS, to quantify trade-offs.
• A simulated two-tiered crawl is validated using real data, showing improved performance and completeness over single-crawler strategies.

Experimental results

Research questions

• RQ1 How does the inclusion of PhantomJS in the crawling pipeline affect frontier size and crawl speed compared to Heritrix and wget?
• RQ2 Can a classifier accurately predict which resources contain deferred representations to enable selective use of PhantomJS?
• RQ3 What is the performance impact of using PhantomJS only on deferred representations versus using it on all resources?
• RQ4 How much larger is the frontier size when combining PhantomJS and Heritrix in a tiered approach compared to using Heritrix alone?
• RQ5 To what extent do duplicate URIs (e.g., session-specific parameters) inflate the frontier when using PhantomJS?

Key findings

• PhantomJS discovered 531,484 URIs—1.75 times more than Heritrix and 4.11 times more than wget—due to its ability to execute JavaScript and uncover dynamically loaded content.
• Heritrix crawled at 2.065 URIs per second, 12.15 times faster than PhantomJS and 2.4 times faster than wget, establishing it as the fastest crawler.
• The two-tiered approach, using PhantomJS only for deferred representations and Heritrix for non-deferred ones, achieved a 5.2× speedup over pure PhantomJS crawling.
• The frontier size in the two-tiered approach was 1.8 times larger than with Heritrix alone, indicating significantly improved URI discovery.
• PhantomJS found 19.70 more embedded resources per URI on average than Heritrix, highlighting its superiority in capturing dynamic content.
• 53% of URIs discovered by PhantomJS were duplicates when session-specific parameters were removed, indicating a need for deduplication in the crawling pipeline.