[Paper Review] On Achieving Privacy-Preserving State-of-the-Art Edge Intelligence

Chabal, Daphnee, Sapra, Dolly+1 more|arXiv (Cornell University)|Feb 10, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper advocates for secret sharing as the most suitable privacy-preserving technique for state-of-the-art edge intelligence, identifying model compression—particularly knowledge distillation and low-rank approximation—as essential enablers to reduce communication and computation overhead in secret-sharing-based DNN inference on modern, complex models like Transformers.

ABSTRACT

Deep Neural Network (DNN) Inference in Edge Computing, often called Edge Intelligence, requires solutions to insure that sensitive data confidentiality and intellectual property are not revealed in the process. Privacy-preserving Edge Intelligence is only emerging, despite the growing prevalence of Edge Computing as a context of Machine-Learning-as-a-Service. Solutions are yet to be applied, and possibly adapted, to state-of-the-art DNNs. This position paper provides an original assessment of the compatibility of existing techniques for privacy-preserving DNN Inference with the characteristics of an Edge Computing setup, highlighting the appropriateness of secret sharing in this context. We then address the future role of model compression methods in the research towards secret sharing on DNNs with state-of-the-art performance.

Motivation & Objective

• Address the gap in privacy-preserving DNN inference solutions tailored for modern, computationally complex edge AI workloads.
• Assess the compatibility of existing privacy-preserving techniques with the constraints of edge computing environments, including limited device resources and real-time requirements.
• Identify secret sharing as the most appropriate technique for edge intelligence due to its strong security guarantees and alignment with decentralized, low-trust edge architectures.
• Explore how model compression techniques can reduce the computational and communication burden of secret-sharing-based DNN inference, enabling deployment on state-of-the-art models.
• Provide research guidance for integrating secret sharing with emerging DNN architectures such as Transformers, which feature novel layers like Self-Attention that current protocols do not fully support.

Proposed method

• Evaluate existing privacy-preserving techniques for DNN inference in edge settings, focusing on their scalability and compatibility with real-world, state-of-the-art models.
• Identify secret sharing as the most viable approach due to its ability to protect model weights, inputs, and outputs without requiring trust in edge servers.
• Analyze the computational cost of secret sharing in edge environments, particularly the high number of non-linear operations and communication rounds in multi-party settings.
• Investigate three model compression strategies—quantization, pruning, knowledge distillation, and low-rank approximation—for reducing communication and computation in secret-sharing pipelines.
• Propose that knowledge distillation and low-rank approximation are most effective for secret-sharing-based inference, as they reduce both model size and the number of layers and operations.
• Highlight the need for new secret-sharing protocols that can handle non-uniform, complex layers such as Self-Attention and recurrent units, which are common in modern Transformers.

Experimental results

Research questions

• RQ1 How compatible are existing privacy-preserving DNN inference techniques with the computational and communication constraints of edge computing environments?
• RQ2 Why is secret sharing uniquely suited for privacy-preserving edge intelligence compared to other cryptographic techniques?
• RQ3 To what extent can model compression techniques reduce the performance overhead of secret-sharing-based DNN inference in edge settings?
• RQ4 What are the key challenges in adapting secret-sharing protocols to state-of-the-art DNNs with novel layers such as Self-Attention and Transformers?
• RQ5 Which model compression method—quantization, pruning, knowledge distillation, or low-rank approximation—most effectively enables secure, efficient secret-sharing inference on modern models?

Key findings

• Secret sharing is the most appropriate privacy-preserving technique for edge intelligence due to its strong security guarantees and compatibility with decentralized, low-trust edge deployments.
• Current secret-sharing protocols struggle with state-of-the-art DNNs due to high communication rounds and computational overhead, especially from non-linear operations and complex layers like Self-Attention.
• Quantization reduces communication volume but does not reduce the number of communication rounds, limiting its effectiveness in high-latency edge settings.
• Pruning offers limited benefit for secret-sharing overhead, as it does not reduce the number of communication rounds or the complexity of non-linear operations.
• Knowledge distillation and low-rank approximation are more promising because they reduce the number of layers and operations, thereby decreasing both computation and communication requirements per layer.
• Recent successes in compressing models like BERT into DistilBERT and Ladabert demonstrate that compressed models can maintain high accuracy while being more amenable to secret-sharing-based inference.