[Paper Review] Privacy Preserving Prompt Engineering: A Survey

Kennedy Edemacu, Xintao Wu|arXiv (Cornell University)|Apr 9, 2024

Cryptography and Data Security4 citations

TL;DR

This survey systematically categorizes and analyzes privacy-preserving prompting techniques for large language models (LLMs), focusing on in-context learning (ICL) and prompting. It evaluates methods based on differential privacy, data obfuscation, federated learning, and cryptographic techniques, highlighting trade-offs in privacy, utility, and fairness, and identifies key research gaps in trustworthiness, multimodal extensions, and benchmarking.

ABSTRACT

Pre-trained language models (PLMs) have demonstrated significant proficiency in solving a wide range of general natural language processing (NLP) tasks. Researchers have observed a direct correlation between the performance of these models and their sizes. As a result, the sizes of these models have notably expanded in recent years, persuading researchers to adopt the term large language models (LLMs) to characterize the larger-sized PLMs. The size expansion comes with a distinct capability called in-context learning (ICL), which represents a special form of prompting and allows the models to be utilized through the presentation of demonstration examples without modifications to the model parameters. Although interesting, privacy concerns have become a major obstacle in its widespread usage. Multiple studies have examined the privacy risks linked to ICL and prompting in general, and have devised techniques to alleviate these risks. Thus, there is a necessity to organize these mitigation techniques for the benefit of the community. This survey provides a systematic overview of the privacy protection methods employed during ICL and prompting in general. We review, analyze, and compare different methods under this paradigm. Furthermore, we provide a summary of the resources accessible for the development of these frameworks. Finally, we discuss the limitations of these frameworks and offer a detailed examination of the promising areas that necessitate further exploration.

Motivation & Objective

• To systematically organize and analyze existing privacy-preserving prompting frameworks for LLMs, particularly in the context of in-context learning (ICL).
• To classify privacy mechanisms by their privacy models, objectives, and threat models, distinguishing non-DP, local DP, global DP, and other scenarios.
• To evaluate the impact of privacy-preserving techniques on LLM trustworthiness facets beyond privacy, including fairness, robustness, safety, and truthfulness.
• To identify critical research gaps in benchmark datasets, open-source tools, and integration into real-world systems.
• To explore the extension of privacy-preserving prompting to multimodal models such as vision-language models (VLMs).

Proposed method

• Categorizing privacy-preserving prompting frameworks into four main groups: non-differential privacy (non-DP), local differential privacy (local DP), global differential privacy (global DP), and other scenarios.
• Analyzing privacy mechanisms such as differential privacy (DP), data sanitization, lattice-based obfuscation, encryption, and model ensembling for their privacy guarantees and operational constraints.
• Mapping each framework to specific privacy targets, including model input, output, and intermediate representations, using a mechanism-target linkage model.
• Evaluating the performance and privacy trade-offs of each method, particularly the negative impact of DP on small-class data in synthetic data generation.
• Surveying existing open-source libraries and software tools, noting their reliance on anonymization, which is vulnerable to re-identification attacks.
• Proposing future research directions, including secure integration of user-controlled generation control and extending DP-forward techniques to vision, audio, and video models.

Experimental results

Research questions

• RQ1 How do different privacy-preserving prompting frameworks compare in terms of privacy guarantees, utility, and computational overhead?
• RQ2 What are the trade-offs between privacy, fairness, and model performance when applying differential privacy to in-context learning?
• RQ3 How do existing frameworks address the risk of server-side compromise when sharing generation control with users?
• RQ4 To what extent can existing privacy-preserving techniques be extended to multimodal models such as vision-language models (VLMs)?
• RQ5 What are the limitations of current benchmark datasets and open-source tools in enabling reproducible and standardized evaluation of privacy-preserving prompting?

Key findings

• Differential privacy mechanisms, while effective for privacy, can lead to unfairness in synthetic data generation, particularly for small-sized classes.
• Many existing open-source privacy-preserving frameworks rely on anonymization, which is known to be vulnerable to re-identification attacks.
• Cryptographic and lattice-based frameworks may introduce security risks to the server if generation control is shared with untrusted clients.
• The integration of privacy-preserving prompting into real-world systems is hindered by the lack of standardized benchmark datasets and production-ready open-source libraries.
• Extending privacy-preserving techniques like DP-forward to vision, audio, and video models is a promising but underexplored research direction.
• There is a critical lack of studies evaluating the impact of privacy-preserving prompting on broader LLM trustworthiness dimensions such as truthfulness, safety, and robustness.