[Paper Review] A Cost-based Storage Format Selector for Materialization in Big Data Frameworks

Rana Faisal Munir, Alberto Abelló+3 more|ArXiv.org|Jun 11, 2018

Cloud Computing and Resource Management22 references3 citations

TL;DR

This paper proposes a cost-based storage format selector that dynamically chooses the optimal data layout—horizontal, vertical, or hybrid—for materialized results in big data workflows. By analyzing access patterns and using a generic cost model instantiated on HDFS with SequenceFile, Avro, and Parquet, it achieves up to 25% average performance gain over fixed-format solutions.

ABSTRACT

Modern big data frameworks (such as Hadoop and Spark) allow multiple users to do large-scale analysis simultaneously. Typically, users deploy Data-Intensive Workflows (DIWs) for their analytical tasks. These DIWs of different users share many common parts (i.e, 50-80%), which can be materialized to reuse them in future executions. The materialization improves the overall processing time of DIWs and also saves computational resources. Current solutions for materialization store data on Distributed File Systems (DFS) by using a fixed data format. However, a fixed choice might not be the optimal one for every situation. For example, it is well-known that different data fragmentation strategies (i.e., horizontal, vertical or hybrid) behave better or worse according to the access patterns of the subsequent operations. In this paper, we present a cost-based approach which helps deciding the most appropriate storage format in every situation. A generic cost-based storage format selector framework considering the three fragmentation strategies is presented. Then, we use our framework to instantiate cost models for specific Hadoop data formats (namely SequenceFile, Avro and Parquet), and test it with realistic use cases. Our solution gives on average 33% speedup over SequenceFile, 11% speedup over Avro, 32% speedup over Parquet, and overall, it provides upto 25% performance gain.

Motivation & Objective

• To address the limitation of fixed storage formats in big data frameworks that degrade performance due to mismatched layouts.
• To reduce materialized result load times by selecting optimal fragmentation strategies (horizontal, vertical, hybrid) based on subsequent query patterns.
• To design a generic, extensible cost model for I/O operations across different storage formats in Hadoop environments.
• To evaluate the framework on real-world decision support benchmarks, demonstrating consistent performance gains over static formats.

Proposed method

• Develops a generic I/O cost model estimating read and write costs for horizontal, vertical, and hybrid data fragmentation strategies.
• Instantiates the cost model for three Hadoop-native formats: SequenceFile, Avro, and Parquet, using statistical metadata from query workloads.
• Implements a hybrid selector framework that uses rule-based heuristics for cold starts and transitions to cost-based selection once statistics are collected.
• Employs a two-phase approach: initial rule-based layout selection with metadata collection, followed by cost-model-driven format selection.
• Uses real-world DSS benchmarks (TPC-H and TPC-DS) to evaluate performance across diverse workloads.
• Applies cost model parameters derived from actual I/O measurements and query access patterns to guide format selection.

Experimental results

Research questions

• RQ1 How can a cost-based approach improve the performance of materialized results in big data workflows by selecting optimal storage formats?
• RQ2 What are the key cost factors influencing I/O performance across horizontal, vertical, and hybrid data layouts in HDFS?
• RQ3 To what extent can dynamic format selection outperform fixed-format storage in real-world analytical workloads?
• RQ4 How does the cost model’s accuracy and performance vary across different data formats (SequenceFile, Avro, Parquet) and workloads?
• RQ5 Can a hybrid rule-based and cost-based strategy effectively balance cold-start overhead with long-term performance gains?

Key findings

• The proposed system achieves an average 33% speedup over SequenceFile, 11% over Avro, and 32% over Parquet in materialized result loading.
• Overall, the framework delivers up to 25% average performance gain compared to any single fixed storage format.
• The cost model effectively identifies optimal layouts based on access patterns, reducing I/O overhead in data-intensive workflows.
• The transition from rule-based to cost-based selection significantly improves format choice accuracy after initial metadata collection.
• The framework demonstrates consistent performance gains across diverse analytical workloads, including TPC-H and TPC-DS benchmarks.
• Dynamic layout selection outperforms static formats even when the latter are manually tuned for specific workloads.