[Paper Review] High Performance Computing Evaluation A methodology based on Scientific Application Requirements

Mariza Ferro, Antonio Roberto Mury+2 more|arXiv (Cornell University)|Dec 3, 2014

Parallel Computing and Optimization Techniques19 references3 citations

TL;DR

This paper proposes a systematic methodology for evaluating high-performance computing (HPC) systems based on scientific application requirements. It integrates application characteristics, problem size, programming language, and hardware architecture to identify optimal performance configurations, demonstrating that performance peaks when all four factors are co-optimized in a tailored evaluation framework.

ABSTRACT

High Performance Distributed Computing is essential to boost scientific progress in many areas of science and to efficiently deploy a number of complex scientific applications. These applications have different characteristics that require distinct computational resources too. In this work we propose a systematic performance evaluation methodology. The focus of our methodology begins on scientific application characteristics, and then considers how these characteristics interact with the problem size, with the programming language and finally with a specific computational architecture. The computational experiments developed highlight this model of evaluation and indicate that optimal performance is found when we evaluate a combination of application class, program language, problem size and architecture model.

Motivation & Objective

• To address the challenge of selecting optimal HPC configurations for diverse scientific applications.
• To develop a performance evaluation framework that moves beyond hardware-centric metrics toward application-driven criteria.
• To investigate how application characteristics, problem size, programming language, and architecture jointly influence HPC performance.
• To provide a repeatable, systematic methodology for HPC system evaluation that reflects real-world scientific workloads.

Proposed method

• The methodology begins by analyzing scientific application characteristics such as data access patterns, algorithmic complexity, and parallelism requirements.
• It models the interaction between application classes and problem sizes to assess computational demand and scalability.
• The approach incorporates programming language choices (e.g., C, Fortran, Python) as a variable affecting performance due to runtime and compilation overheads.
• It evaluates performance across different computational architectures (e.g., multi-core, GPU, cluster systems) under controlled experimental conditions.
• The framework uses empirical benchmarking with real scientific applications to measure execution time, resource utilization, and scalability.
• Results are synthesized into a performance model that identifies optimal combinations of application, language, problem size, and architecture.

Experimental results

Research questions

• RQ1 How do scientific application characteristics influence the choice of optimal HPC architecture?
• RQ2 What is the impact of programming language on HPC performance for different application classes?
• RQ3 How does problem size affect the performance and scalability of scientific applications across different architectures?
• RQ4 What combination of application class, programming language, problem size, and architecture yields maximum performance?

Key findings

• Optimal HPC performance is achieved only when application characteristics, problem size, programming language, and architecture are evaluated together as an integrated system.
• The choice of programming language significantly affects performance, with lower-level languages like C and Fortran outperforming higher-level languages like Python for compute-intensive tasks.
• Performance varies non-linearly with problem size, and scalability depends on both algorithmic structure and underlying hardware.
• The methodology successfully identifies performance bottlenecks and optimal configurations across diverse scientific workloads.
• The evaluation model demonstrates that hardware-centric benchmarks alone fail to predict real application performance.