[Paper Review] Efficient Deep Learning on Multi-Source Private Data

Nick Hynes, Raymond Cheng+1 more|arXiv (Cornell University)|Jul 17, 2018

Privacy-Preserving Technologies in DataComputer Science28 references78 citations

TL;DR

Myelin is a deep learning framework that enables fully private, differential-privacy-preserving training on multi-source private data using trusted hardware enclaves, achieving competitive performance with non-private CPU training.

ABSTRACT

Machine learning models benefit from large and diverse datasets. Using such datasets, however, often requires trusting a centralized data aggregator. For sensitive applications like healthcare and finance this is undesirable as it could compromise patient privacy or divulge trade secrets. Recent advances in secure and privacy-preserving computation, including trusted hardware enclaves and differential privacy, offer a way for mutually distrusting parties to efficiently train a machine learning model without revealing the training data. In this work, we introduce Myelin, a deep learning framework which combines these privacy-preservation primitives, and use it to establish a baseline level of performance for fully private machine learning.

Motivation & Objective

• Motivate the need for privacy-preserving ML when data from multiple private sources is valuable but sensitive.
• Introduce a system (Myelin) that combines trusted hardware enclaves with differential privacy and data-oblivious computation.
• Demonstrate baseline performance for fully private training on practical CNN models.
• Showcase how TVM-generated, enclave-aware libraries enable efficient private training on commodity hardware.

Proposed method

• Utilizes trusted hardware enclaves (e.g., SGX) to isolate training data and model parameters.
• Applies differential privacy to SGD by clipping per-example gradients and adding Gaussian noise via the Gaussian mechanism with the moments accountant.
• Implements data-oblivious algorithms and fixed-size data chunking to mitigate side-channel and timing leaks.
• Leverages TVM to generate minimal, fused operator libraries for enclave efficiency and reduces the TCB.
• Schedules computation to maximize parallelism and fuses privacy-related steps (clip, noise, sum) to hide latency.
• Supports distributed, enclave-based training with multi-threading to exploit commodity hardware.

Experimental results

Research questions

• RQ1 Can fully private training using trusted hardware achieve practical performance for deep learning models on multi-provider private data?
• RQ2 What is the accuracy vs. speed trade-off when applying differential privacy and data-oblivious methods inside enclaves?
• RQ3 How does Myelin compare to cryptographic or multi-enclave approaches in terms of throughput and model accuracy?
• RQ4 What architectural choices (e.g., TVM-generated operators, per-example gradient handling) best support DP and data-obliviousness in practice?

Key findings

• Fully private Myelin training on VGG9 and ResNet-32 is competitive with non-private CPU training in speed and accuracy, thanks to DP, data-obliviousness, and optimized enclave UX.
• A single Myelin enclave can outperform a distributed setup of four Chiron enclaves in certain configurations, illustrating enclave-based efficiency gains.
• On CIFAR-10, private training with Myelin achieves test accuracies close to non-private baselines (e.g., VGG9 84.4% private vs 89.5% non-private; ResNet-32 90.8% private vs 92.4% non-private).
• Inference with privately trained MobileNet within Myelin matches privacy-preserving inference baselines like Slalom in speed, with comparable accuracy.
• Compared to cryptographic approaches (e.g., Gazelle) and GPU-augmented schemes (Slalom), Myelin shows favorable speed/accuracy trade-offs within a single enclave.
• The results establish a baseline for fully private ML performance using hardware enclaves on commodity hardware.