[Paper Review] Navigating the Data Lake with Datamaran: Automatically Extracting Structure from Log Datasets

Yihan Gao, Silu Huang+1 more|arXiv (Cornell University)|Aug 29, 2017

Web Data Mining and Analysis27 references4 citations

TL;DR

Datamaran is an unsupervised tool that automatically extracts structured relational data from semi-structured log datasets by identifying field and record boundaries without requiring prior knowledge of record structure. It achieves 95.5% extraction accuracy on GitHub log datasets—66% higher than prior unsupervised methods—by combining template generation, assimilation scoring, and iterative refinement to handle multi-line records and noisy formatting.

ABSTRACT

Organizations routinely accumulate semi-structured log datasets generated as the output of code; these datasets remain unused and uninterpreted, and occupy wasted space - this phenomenon has been colloquially referred to as "data lake" problem. One approach to leverage these semi-structured datasets is to convert them into a structured relational format, following which they can be analyzed in conjunction with other datasets. We present Datamaran, an tool that extracts structure from semi-structured log datasets with no human supervision. Datamaran automatically identifies field and record endpoints, separates the structured parts from the unstructured noise or formatting, and can tease apart multiple structures from within a dataset, in order to efficiently extract structured relational datasets from semi-structured log datasets, at scale with high accuracy. Compared to other unsupervised log dataset extraction tools developed in prior work, Datamaran does not require the record boundaries to be known beforehand, making it much more applicable to the noisy log files that are ubiquitous in data lakes. Datamaran can successfully extract structured information from all datasets used in prior work, and can achieve 95% extraction accuracy on automatically collected log datasets from GitHub - a substantial 66% increase of accuracy compared to unsupervised schemes from prior work. Our user study further demonstrates that the extraction results of Datamaran are closer to the desired structure than competing algorithms.

Motivation & Objective

• Address the 'data lake' problem where vast volumes of semi-structured log datasets remain unused due to lack of structure.
• Overcome the limitations of existing unsupervised log extraction tools that require known record boundaries or assume single-line records.
• Enable fully automated, scalable, and accurate transformation of raw log files into structured relational formats for downstream analysis.
• Provide a robust solution that handles multi-line records, noisy formatting, and heterogeneous structures within a single dataset.
• Demonstrate superior performance and usability compared to existing tools like RecordBreaker and prior unsupervised wrappers in both quantitative benchmarks and user studies.

Proposed method

• Formalize structure extraction as an optimization problem using a regularity score function aligned with human judgment.
• Generate candidate structure templates by identifying recurring patterns that satisfy a minimum coverage threshold across log lines.
• Apply a pruning step using an assimilation score function to rank and filter the most coherent and consistent templates.
• Perform iterative evaluation and refinement of remaining templates to improve structural accuracy and consistency.
• Leverage line-level analysis to detect field and record boundaries without relying on external chunking or tagging.
• Support multi-structure detection within a single dataset, enabling extraction of multiple distinct record types from the same log file.

Experimental results

Research questions

• RQ1 Can an unsupervised system automatically extract structured relational data from semi-structured log files without requiring prior knowledge of record boundaries?
• RQ2 How effective is Datamaran in extracting structure from real-world log datasets with complex, multi-line, and noisy formats compared to existing unsupervised tools?
• RQ3 To what extent does Datamaran’s output align with human-expected data structures, as validated through user studies?
• RQ4 How robust is Datamaran to parameter variations and diverse log formats found in enterprise data lakes?
• RQ5 Can Datamaran detect and extract multiple distinct data structures within a single log file, enabling more comprehensive data discovery?

Key findings

• Datamaran achieved 95.5% extraction accuracy on a large collection of GitHub log datasets, representing a 66% improvement over prior unsupervised methods.
• The tool successfully extracted structure from all representative datasets used in prior work, demonstrating broad applicability.
• In a user study, all six participants preferred Datamaran’s output over both RecordBreaker and the raw log files, indicating strong alignment with human expectations.
• Datamaran outperformed RecordBreaker, which achieved only 29.2% extraction accuracy on the same GitHub dataset collection.
• The system is robust to parameter choices, maintaining high performance across diverse log formats and noise levels.
• Datamaran uniquely supports the detection of multiple distinct data structures within a single log file, a capability missing in prior tools.