[Paper Review] A Survey on Preprocessing Methods for Web Usage Data

V. Chitraa, Antony Selvdoss Davamani|arXiv (Cornell University)|Apr 8, 2010

Data Mining Algorithms and Applications5 references84 citations

TL;DR

This paper surveys preprocessing techniques for web usage data, focusing on session reconstruction and noise handling in web log files. It evaluates methods for transforming raw, noisy logs into structured session-level data, enabling effective web usage mining for applications like personalization and adaptive web design.

ABSTRACT

World Wide Web is a huge repository of web pages and links. It provides abundance of information for the Internet users. The growth of web is tremendous as approximately one million pages are added daily. Users' accesses are recorded in web logs. Because of the tremendous usage of web, the web log files are growing at a faster rate and the size is becoming huge. Web data mining is the application of data mining techniques in web data. Web Usage Mining applies mining techniques in log data to extract the behavior of users which is used in various applications like personalized services, adaptive web sites, customer profiling, prefetching, creating attractive web sites etc., Web usage mining consists of three phases preprocessing, pattern discovery and pattern analysis. Web log data is usually noisy and ambiguous and preprocessing is an important process before mining. For discovering patterns sessions are to be constructed efficiently. This paper reviews existing work done in the preprocessing stage. A brief overview of various data mining techniques for discovering patterns, and pattern analysis are discussed. Finally a glimpse of various applications of web usage mining is also presented.

Motivation & Objective

• To analyze and categorize existing preprocessing techniques for web usage data to improve data quality before mining.
• To identify challenges in handling noisy and ambiguous web log data due to the high volume and complexity of web traffic.
• To provide a foundation for effective session reconstruction, a critical step in web usage mining.
• To support downstream applications such as personalization, customer profiling, and adaptive web systems by improving data preparation.
• To offer a comprehensive overview of preprocessing methods, including sessionization, data cleaning, and normalization techniques.

Proposed method

• Surveying and classifying existing preprocessing methods for web usage data, particularly focusing on session reconstruction from raw web logs.
• Analyzing techniques for handling noise, such as filtering out bot traffic and correcting inconsistent timestamps.
• Evaluating sessionization algorithms that group user requests into logical sessions based on time gaps and user identifiers.
• Reviewing normalization methods to standardize user agent strings, URLs, and other attributes for consistent analysis.
• Comparing state-of-the-art approaches in terms of accuracy, efficiency, and scalability on large-scale web log datasets.
• Providing a framework for selecting preprocessing techniques based on data characteristics and target applications.

Experimental results

Research questions

• RQ1 What are the primary challenges in preprocessing raw web log data for usage mining?
• RQ2 How do different sessionization techniques handle time gaps and user session boundaries?
• RQ3 What methods are effective in reducing noise and improving data quality in web logs?
• RQ4 How do preprocessing choices impact the accuracy and efficiency of subsequent pattern discovery in web usage mining?
• RQ5 What are the trade-offs between scalability and precision in preprocessing web usage data?

Key findings

• Preprocessing is a critical and non-trivial step in web usage mining, significantly affecting downstream analysis quality.
• Session reconstruction remains a major challenge due to inconsistent logging practices and lack of standardized session boundaries.
• Noise reduction techniques, such as bot detection and log filtering, improve data quality and reduce false patterns.
• Normalization of URLs and user agent strings enhances consistency and enables more accurate user behavior analysis.
• The choice of preprocessing method directly influences the performance and reliability of pattern discovery in web usage mining.
• No single preprocessing method is universally optimal; selection depends on data characteristics and application goals.