[Paper Review] Utilizing the World Wide Web as an Encyclopedia: Extracting Term Descriptions from Semi-Structured Texts

Atsushi Fujii, Tetsuya Ishikawa|ArXiv.org|Nov 2, 2000

Natural Language Processing Techniques6 references4 citations

TL;DR

This paper proposes a method to automatically extract and summarize technical term descriptions from the World Wide Web using linguistic patterns and HTML structure. By combining NLP-based and HTML-based extraction, language modeling for filtering, and hierarchical Bayesian clustering for summarization, the system achieves 67.9% accuracy in identifying relevant descriptions, enabling users to understand terms through just two representative descriptions on average.

ABSTRACT

In this paper, we propose a method to extract descriptions of technical terms from Web pages in order to utilize the World Wide Web as an encyclopedia. We use linguistic patterns and HTML text structures to extract text fragments containing term descriptions. We also use a language model to discard extraneous descriptions, and a clustering method to summarize resultant descriptions. We show the effectiveness of our method by way of experiments.

Motivation & Objective

• To address the limitation of traditional encyclopedias, which are expensive to produce and often outdated, by leveraging the dynamic and vast content of the World Wide Web as a real-time source of term descriptions.
• To automatically extract accurate and meaningful descriptions of technical terms from semi-structured web pages, overcoming the noise and redundancy inherent in raw web content.
• To improve usability by summarizing multiple extracted descriptions into distinct clusters representing different viewpoints or word senses, reducing redundancy.
• To evaluate the effectiveness of combining linguistic patterns, HTML structure, language modeling, and clustering in a practical, scalable framework for web-based encyclopedia construction.

Proposed method

• The system uses NLP-based linguistic patterns and HTML-based structural heuristics to identify candidate text fragments containing term descriptions from web pages.
• Preprocessing removes irrelevant elements such as HTML tags, extra whitespace, and non-linguistic content like email addresses to standardize input.
• A trigram-based language model filters out non-linguistic or noisy fragments by measuring their linguistic plausibility, retaining only those likely to be meaningful descriptions.
• Extracted descriptions for a single term are clustered using the HBC (Hierarchical Bayesian Clustering) method to group similar descriptions and identify representative ones.
• The system supports both background indexing (periodic updates) and dynamic on-demand extraction, allowing users to access descriptions in real time.
• Representative descriptions from each cluster are presented to users, ensuring coverage of different perspectives and word senses while minimizing redundancy.

Experimental results

Research questions

• RQ1 Can linguistic and structural patterns in HTML be effectively combined to extract term descriptions from semi-structured web pages?
• RQ2 To what extent does a language model improve the accuracy of description extraction by filtering out non-linguistic or irrelevant fragments?
• RQ3 Can clustering of extracted descriptions effectively group similar descriptions and represent distinct viewpoints or word senses?
• RQ4 What is the overall accuracy of the system in retrieving useful, understandable descriptions for technical terms?

Key findings

• The NLP/HTML-based method achieved an initial extraction accuracy of 63.5%, demonstrating the feasibility of using pattern-based extraction from web content.
• Incorporating a trigram-based language model improved the accuracy to 67.9%, reducing noise and enhancing the quality of extracted descriptions.
• The HBC clustering method successfully summarized descriptions, with 66.7% of the presented representative descriptions being correct, indicating effective grouping by viewpoint or sense.
• The system correctly identified polysemy in the term 'korokeishon' (collocation), with two of three representative descriptions corresponding to 'word collocations' and one to 'position of machinery', showing sensitivity to multiple meanings.
• On average, only two descriptions per term are sufficient for users to understand a term, as confirmed by the evaluation of representative descriptions.
• The method is effective for both static and dynamic retrieval, supporting real-time access to term descriptions without relying on pre-indexed databases.