[Paper Review] CryptMPI: A Fast Encrypted MPI Library

Abu Naser, Cong Wu+5 more|arXiv (Cornell University)|Oct 13, 2020

Cryptography and Data Security15 references4 citations

TL;DR

CryptMPI is a high-performance encrypted MPI library that accelerates secure message-passing in HPC by leveraging multi-threaded pipelined encryption and adaptive runtime parameter tuning. It reduces encryption overheads to under 6% for large messages on high-speed networks, outperforming naive encryption and IPSec by up to 3x in throughput and minimizing performance degradation even under heavy load.

ABSTRACT

The cloud infrastructure must provide security for High-Performance Computing (HPC) applications of sensitive data to execute in such an environment. However, supporting security in the communication infrastructure of today's public cloud is challenging, because current networks for data centers are so fast that adding encryption can incur very significant overheads. In this work, we introduce CryptMPI, a high performance encrypted MPI library that supports communication with both integrity and privacy. We present the techniques in CryptMPI and report our benchmarking results using micro-benchmarks and NAS parallel benchmarks. The evaluation results indicate that the aforementioned techniques are effective in improving the performance of encrypted communication.

Motivation & Objective

• To address the critical performance bottleneck of encryption in modern high-speed data center networks for HPC workloads.
• To provide strong security guarantees—confidentiality and integrity—without sacrificing communication performance in MPI-based applications.
• To overcome the limitations of existing solutions, such as weak encryption modes (e.g., ECB) or poor scalability (e.g., IPSec), which severely degrade throughput.
• To exploit underutilized CPU resources, such as hyper-threads, to parallelize encryption and decryption operations in MPI communication.
• To design a system that dynamically tunes thread count and data chunking to minimize end-to-end communication latency.

Proposed method

• CryptMPI uses AES-GCM for authenticated encryption, ensuring both privacy and integrity of MPI messages.
• It enables pipelined encryption and decryption by splitting large messages into chunks and processing them in parallel across multiple hyper-threads.
• The system employs an adaptive performance model to automatically determine optimal runtime parameters, including the number of threads and chunk size, based on measured network and CPU characteristics.
• It leverages Intel AES-NI and CLMUL instructions to accelerate cryptographic operations, minimizing per-operation overhead.
• The library integrates directly into the MPI layer, avoiding reliance on lower-layer protocols like IPSec, which introduce serialization and performance bottlenecks.
• It uses a hybrid approach where encryption is performed in parallel on the sender and receiver, overlapping computation with communication to hide latency.

Experimental results

Research questions

• RQ1 Can a custom-optimized encrypted MPI library achieve significantly lower performance overhead than naive encryption or IPSec in high-speed data center networks?
• RQ2 How effective is multi-threaded pipelined encryption in reducing the latency of large-message MPI communication under real HPC workloads?
• RQ3 To what extent can adaptive parameter tuning of thread count and chunk size improve end-to-end performance across diverse network and computational conditions?
• RQ4 Does the performance benefit of CryptMPI diminish under high computational load or when communication is already overlapped with computation?
• RQ5 Can a system designed for point-to-point communication be extended to efficiently support collective operations in future work?

Key findings

• On a 40Gbps InfiniBand cluster, CryptMPI reduced encryption overhead to just 5.03% for 4MB messages with two concurrent communication pairs, compared to 178.48% for the naive approach.
• For 1MB messages, CryptMPI achieved 1.2 GBps throughput, representing a 60% improvement over the naive AES-GCM method (1.2 GBps vs. 0.72 GBps) and significantly outperforming IPSec, which achieved only ~1 GBps.
• In NAS Parallel Benchmark CG, CryptMPI incurred only 12.3% overhead on inter-node communication time, compared to 79% for the naive method, despite a total execution time overhead of 20.2%.
• For 2D stencil kernels with 60% computational load and 2MB messages, CryptMPI’s encryption overhead was 206%, compared to 331% for the naive approach, demonstrating clear performance gains under moderate load.
• Even under heavy 80% computational load, CryptMPI reduced overhead from 450% (naive) to 384%, showing resilience in performance-critical scenarios.
• The performance model enabled CryptMPI to automatically select optimal configurations—such as 2 threads and 4 chunks for 2MB messages—demonstrating effective runtime adaptation.