[Paper Review] HyPHEN: A Hybrid Packing Method and Optimizations for Homomorphic Encryption-Based Neural Networks

Dong-Hwan Kim, Jaiyoung Park+3 more|arXiv (Cornell University)|Feb 5, 2023

Cryptography and Data Security4 citations

TL;DR

HyPHEN proposes a hybrid packing method and optimized convolution algorithms (RAConv and CAConv) to drastically reduce homomorphic operations in FHE-based CNN inference. By enabling polymorphic data formats and minimizing costly rotations and memory bloat, HyPHEN achieves 1.4 seconds for CIFAR-10 (ResNet-20) and enables the first ImageNet inference (14.7 seconds) with ResNet-18.

ABSTRACT

Convolutional neural network (CNN) inference using fully homomorphic encryption (FHE) is a promising private inference (PI) solution due to the capability of FHE that enables offloading the whole computation process to the server while protecting the privacy of sensitive user data. Prior FHE-based CNN (HCNN) work has demonstrated the feasibility of constructing deep neural network architectures such as ResNet using FHE. Despite these advancements, HCNN still faces significant challenges in practicality due to the high computational and memory overhead. To overcome these limitations, we present HyPHEN, a deep HCNN construction that incorporates novel convolution algorithms (RAConv and CAConv), data packing methods (2D gap packing and PRCR scheme), and optimization techniques tailored to HCNN construction. Such enhancements enable HyPHEN to substantially reduce the memory footprint and the number of expensive homomorphic operations, such as ciphertext rotation and bootstrapping. As a result, HyPHEN brings the latency of HCNN CIFAR-10 inference down to a practical level at 1.4 seconds (ResNet-20) and demonstrates HCNN ImageNet inference for the first time at 14.7 seconds (ResNet-18).

Motivation & Objective

• Address the high computational and memory overhead in FHE-based CNN inference (HCNN), which limits practical deployment.
• Reduce the number of expensive homomorphic operations—especially ciphertext rotation and bootstrapping—during inference.
• Minimize memory footprint caused by plaintext weight expansion, particularly for high-resolution images and large models like ResNet-18 and ResNet-50.
• Enable practical, low-latency private inference for real-world datasets such as ImageNet by optimizing data packing and convolution algorithms.
• Achieve end-to-end private inference with succinct communication and full server-side computation offloading while preserving model privacy.

Proposed method

• Introduce RAConv and CAConv, polymorphic FHE convolution algorithms tailored to different data formats to reduce reliance on costly ciphertext rotations.
• Develop 2D gap packing to efficiently manage spatial gaps introduced by strided convolutions, minimizing rotation overhead during data reorganization.
• Propose the PRCR (Packing with Reuse of Ciphertext Rows) scheme to eliminate redundant duplication of weight elements across input image segments, reducing plaintext memory footprint.
• Use a hybrid data packing strategy that allows dynamic selection of data formats (e.g., dense, row-wise, segmented) per layer to minimize data rearrangement costs.
• Leverage low-degree polynomial approximations for ReLU from AESPA to reduce activation computation cost, avoiding expensive homomorphic operations.
• Optimize data layout and weight reuse across image segments by dividing input images into row-wise segments, aligning weight sizes with segment dimensions to reduce memory expansion.

Experimental results

Research questions

• RQ1 Can a hybrid packing method reduce the number of homomorphic rotations required during FHE-based CNN inference?
• RQ2 How can polymorphic convolution algorithms be designed to adapt to different data formats and minimize data rearrangement costs?
• RQ3 To what extent can memory footprint be reduced in FHE-based CNNs by rethinking weight plaintext layout and reuse strategies?
• RQ4 Can the combination of optimized packing, convolution, and activation methods enable practical inference times for large models like ResNet-18 on ImageNet?
• RQ5 What is the impact of data format consistency and dynamic format switching on the overall performance of HCNN inference?

Key findings

• HyPHEN reduces CIFAR-10 inference latency for ResNet-20 to 1.40 seconds on GPU, a 2.15× improvement over the prior state-of-the-art.
• The first end-to-end FHE-based inference on ImageNet is achieved with ResNet-18 in 14.69 seconds, demonstrating scalability to real-world datasets.
• The PRCR scheme significantly reduces memory footprint by eliminating per-weight-element duplication, especially benefiting large models like ResNet-50.
• RAConv and CAConv reduce the number of homomorphic rotations by eliminating the need for inter-layer data format conversion (IR) and minimizing channel summation (RaS) rotations.
• On CPU, HyPHEN achieves 1.83× and 2.15× lower latency than the prior state-of-the-art for ResNet-20 and ResNet-18, respectively.
• The hybrid packing method with 2D gap handling enables efficient convolution on strided feature maps while minimizing rotation cost and preserving data locality.