[Paper Review] Data lake concept and systems: a survey.

Rihan Hai, Christoph Quix+1 more|arXiv (Cornell University)|Jun 17, 2021

Data Quality and Management107 references21 citations

TL;DR

This survey proposes a comprehensive analysis of data lake concepts and systems to address the challenges of managing diverse, large-volume data in traditional 'schema-on-write' architectures. It classifies existing data lake systems by functionality, provides architectural insights, and identifies open research challenges to guide future development in data lake design and implementation.

ABSTRACT

Although big data has been discussed for some years, it still has many research challenges, especially the variety of data. It poses a huge difficulty to efficiently integrate, access, and query the large volume of diverse data in information silos with the traditional 'schema-on-write' approaches such as data warehouses. Data lakes have been proposed as a solution to this problem. They are repositories storing raw data in its original formats and providing a common access interface. This survey reviews the development, definition, and architectures of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing data lake systems based on their provided functions, which makes this survey a useful technical reference for designing, implementing and applying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey would motivate the future development of data lake research and practice.

Motivation & Objective

• To address the limitations of traditional data warehouses in handling the variety of big data.
• To explore the data lake paradigm as a solution for storing raw, diverse data in its native format.
• To provide a systematic classification of existing data lake systems based on their functional capabilities.
• To identify key research challenges in data lake design, implementation, and deployment.
• To serve as a technical reference for researchers and practitioners building and applying data lake systems.

Proposed method

• The paper conducts a comprehensive survey of data lake development, definitions, and system architectures.
• It classifies data lake systems based on their provided functions, enabling a structured comparison.
• The survey analyzes the evolution of data lake concepts and their technical foundations.
• It evaluates data lake systems using criteria such as data ingestion, storage, query processing, and metadata management.
• The study synthesizes findings from existing literature to highlight architectural patterns and design trade-offs.
• It discusses open research challenges to guide future innovation in data lake technologies.

Experimental results

Research questions

• RQ1 How do data lakes differ from traditional data warehouses in handling data variety and volume?
• RQ2 What are the core architectural components and design principles of modern data lake systems?
• RQ3 How do existing data lake systems classify and manage metadata and data lineage?
• RQ4 What are the key functional capabilities and limitations of current data lake platforms?
• RQ5 What open research challenges remain in data lake design and deployment?

Key findings

• Data lakes provide a common access interface for raw data in its original format, overcoming the rigidity of 'schema-on-write' approaches.
• The survey classifies data lake systems based on their functional features, offering a useful technical reference for system design.
• Data lakes support flexible data integration and querying across heterogeneous data sources and formats.
• The survey identifies significant research gaps in data quality, security, and performance optimization within data lakes.
• Existing systems vary widely in support for metadata management, data lineage, and transactional consistency.
• The paper emphasizes the need for standardized evaluation frameworks and improved tooling to advance data lake research and practice.