[Paper Review] The Klarna Product Page Dataset: Web Element Nomination with Graph Neural Networks and Large Language Models

Alexandra Hotti, Riccardo Sven Risuleo+3 more|arXiv (Cornell University)|Nov 3, 2021

Web Data Mining and Analysis4 citations

TL;DR

This paper introduces the Klarna Product Page Dataset, a large-scale, multilingual collection of 51,701 real e-commerce product pages from 8,175 websites across eight geographic regions, annotated with five key elements per page. It proposes a hybrid method combining a simple GCN for initial element nomination and a zero-shot large language model (LLM) to refine predictions, achieving a 16.8 percentage point accuracy gain without fine-tuning, while also introducing a novel training procedure that further improves performance.

ABSTRACT

Description The Klarna Product Page Dataset is a dataset of publicly available pages corresponding to products sold online on various e-commerce websites. The dataset contains offline snapshots of 51,701 product pages collected from 8,175 distinct merchants across 8 different markets (US, GB, SE, NL, FI, NO, DE, AT) between 2018 and 2019. On each page, analysts labelled 5 elements of interest: the price of the product, its image, its name and the add-to-cart and go-to-cart buttons (if found). These labels are present in the HTML code as an attribute called klarna-ai-label taking one of the values: Price, Name, Main picture, Add to cart and Cart. The snapshots are available in 3 formats: as MHTML files (~24GB), as WebTraversalLibrary (WTL) snapshots (~7.4GB), and as screeshots (~8.9GB). The MHTML format is less lossy, a browser can render these pages though any Javascript on the page is lost. The WTL snapshots are produced by loading the MHTML pages into a chromium-based browser. To keep the WTL dataset compact, the screenshots of the rendered MTHML are provided separately; here we provide the HTML of the rendered DOM tree and additional page and element metadata with rendering information (bounding boxes of elements, font sizes etc.). The folder structure of the screenshot dataset is identical to the one the WTL dataset and can be used to complete the WTL snapshots with image information. For convenience, the datasets are provided with a train/test split in which no merchants in the test set are present in the training set. Corresponding Publication For more information about the contents of the datasets (statistics etc.) please refer to the following TMLR paper. GitHub Repository The code needed to re-run the experiments in the publication accompanying the dataset can be accessed here. Citing If you found this dataset useful in your research, please cite the paper as follows: @article{hotti2024the, title={The Klarna Product Page Dataset: Web Element Nomination with Graph Neural Networks and Large Language Models}, author={Alexandra Hotti and Riccardo Sven Risuleo and Stefan Magureanu and Aref Moradi and Jens Lagergren}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2024}, url={https://openreview.net/forum?id=zz6FesdDbB}, note={} }

Motivation & Objective

• To address the lack of large-scale, realistic, and diverse web page datasets for web element nomination.
• To benchmark graph neural networks (GNNs) on real-world web element nomination tasks.
• To explore the integration of GNNs with large language models (LLMs) for improved nomination accuracy.
• To develop a novel training procedure that enhances nomination performance on complex, real-world webpages.
• To provide a foundation for future research in web automation and LLM-based web interaction.

Proposed method

• The Klarna Product Page Dataset comprises 51,701 real product pages from 8,175 e-commerce websites across eight countries, with five manually labeled elements per page.
• A simple GCN-Mean model is used to generate initial nominations based on local neighborhood information in the DOM tree.
• A subset of high-confidence, relevant elements from GCN predictions is selected and passed to a zero-shot large language model (LLM) for final nomination.
• The method employs a training refinement procedure that identifies and adds hard examples to the training set to improve model confidence and ranking quality.
• A novel Challenge Nomination Training Procedure is introduced to boost nomination accuracy by optimizing for the nomination objective directly.
• The approach leverages rendered page screenshots as a complementary dataset to support future computer vision-based web analysis.

Experimental results

Research questions

• RQ1 Can a simple GCN outperform complex state-of-the-art nomination methods on real-world, diverse webpages?
• RQ2 To what extent can a zero-shot large language model improve nomination accuracy when applied to GCN-predicted candidates without fine-tuning?
• RQ3 How does the timing and frequency of hard example augmentation affect model convergence and nomination performance?
• RQ4 Does a training procedure optimized for the nomination objective (Challenge Nomination Training Procedure) yield measurable improvements over standard training?
• RQ5 How does multilingual and geographically diverse data impact the generalization and robustness of web element nomination models?

Key findings

• A simple GCN-Mean model outperforms more complex state-of-the-art methods, including FreeDOM and DOM-Q-NET, on the Klarna Product Page Dataset.
• The hybrid GCN-LLM approach improves nomination accuracy by 16.8 percentage points without requiring any fine-tuning of the LLM.
• Hard example augmentation during training leads to a steady improvement in confidence gap, indicating better ranking quality and alignment with the nomination objective.
• The Challenge Nomination Training Procedure further boosts nomination accuracy, demonstrating the value of objective-specific training strategies.
• The timing of augmentation has minimal impact on final accuracy, suggesting that early identification of key hard examples significantly improves model performance.
• The dataset’s multilingual and geographically diverse nature enables training on real-world layout and language variations, enhancing model robustness.