[Paper Review] EVEREST: A design environment for extreme-scale big data analytics on heterogeneous platforms

Christian Pilato, Stanislav Böhm+16 more|arXiv (Cornell University)|Jan 1, 2021

Cloud Computing and Resource ManagementComputer Science47 references27 citations

TL;DR

EVEREST presents a holistic, data-driven design environment for extreme-scale big data analytics on heterogeneous, distributed, and secure platforms, leveraging domain-specific languages, high-level synthesis, and virtualized runtime orchestration to enable non-expert developers to automatically generate and deploy optimized hardware-software variants. The key contribution is a unified SDK that improves performance, energy efficiency, and programmability across HPC, cloud, and edge systems through AI-accelerated compilation and dynamic resource adaptation.

ABSTRACT

High-Performance Big Data Analytics (HPDA) applications are characterized by huge volumes of distributed and heterogeneous data that require efficient computation for knowledge extraction and decision making. Designers are moving towards a tight integration of computing systems combining HPC, Cloud, and IoT solutions with artificial intelligence (AI). Matching the application and data requirements with the characteristics of the underlying hardware is a key element to improve the predictions thanks to high performance and better use of resources. We present EVEREST, a novel H2020 project started on October 1, 2020, that aims at developing a holistic environment for the co-design of HPDA applications on heterogeneous, distributed, and secure platforms. EVEREST focuses on programmability issues through a data-driven design approach, the use of hardware-accelerated AI, and an efficient runtime monitoring with virtualization support. In the different stages, EVEREST combines state-of-the-art programming models, emerging communication standards, and novel domain-specific extensions. We describe the EVEREST approach and the use cases that drive our research.

Motivation & Objective

• Address the growing challenge of programming extreme-scale, heterogeneous big data analytics (HPDA) systems that combine HPC, cloud, and edge resources with AI workloads.
• Overcome programmability barriers in HPDA by abstracting low-level hardware complexity for non-expert developers through domain-specific languages and high-level synthesis.
• Enable efficient, secure, and scalable data processing by co-designing applications with hardware characteristics, including memory hierarchy, communication patterns, and security constraints.
• Improve performance and energy efficiency through dynamic selection of code and hardware variants at runtime, based on workload and data conditions.
• Support secure data processing across distributed systems using hardware-assisted encryption, access control, and side-channel protection mechanisms.

Proposed method

• Employ a data-centric, domain-specific language (DSL) framework to express application semantics, security requirements, and data access patterns in high-level abstractions.
• Integrate state-of-the-art programming models (SYCL, OpenCL, OpenMP) with novel domain-specific extensions to enable algorithmic and data-aware optimizations.
• Use high-level synthesis (HLS) to generate custom FPGA accelerators from high-level C/C++ or AI framework code, enabling hardware acceleration for data-intensive workloads.
• Implement a multi-variant compilation framework that explores alternative software and hardware code paths, guided by performance, memory, and energy metrics.
• Build a virtualized runtime environment that abstracts underlying CPU and accelerator heterogeneity (x86, ARM, RISC-V, FPGAs) to enable portable, dynamic execution.
• Apply unified intermediate representation (IR) and metadata-driven compilation to co-optimize computation, communication, and storage, moving computation closer to data.

Experimental results

Research questions

• RQ1 How can a unified design environment be created to co-optimize HPDA applications across heterogeneous, distributed, and secure platforms?
• RQ2 To what extent can domain-specific languages and high-level synthesis improve programmability and performance portability for non-expert developers in extreme-scale data analytics?
• RQ3 How can dynamic runtime adaptation of code and hardware variants improve performance and energy efficiency under changing workloads and data conditions?
• RQ4 What role does hardware-assisted data protection play in securing AI-driven big data analytics across edge and cloud deployments?
• RQ5 How can a data-driven compilation framework effectively balance functional requirements, non-functional constraints (latency, energy), and hardware characteristics?

Key findings

• The EVEREST SDK enables non-expert developers to express complex HPDA workloads using domain-specific languages, significantly reducing the effort required to target heterogeneous FPGA-based platforms.
• By leveraging high-level synthesis and multi-variant compilation, the framework generates optimized hardware accelerators that improve performance and reduce energy consumption in data-intensive workloads.
• Runtime virtualization allows transparent selection of optimal code and hardware variants, enabling dynamic adaptation to workload and data changes across diverse CPU and accelerator architectures.
• Hardware-assisted encryption and information flow tracking are integrated into the platform to protect data confidentiality, integrity, and prevent side-channel attacks in distributed deployments.
• The framework reduces overall workflow processing time and improves resource utilization in large-scale, distributed environments through intelligent memory management and data-locality optimization.
• Use cases in weather forecasting, air-quality monitoring, and intelligent transportation demonstrate that the SDK enables faster, more accurate predictions through real-time and historical data fusion with AI.