[Paper Review] PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models

Chenhao Tang, Zhengliang Liu+9 more|arXiv (Cornell University)|Sep 19, 2023

Privacy-Preserving Technologies in DataComputer Science16 citations

TL;DR

PolicyGPT uses large language models in a zero-shot framework to categorize privacy-policy text, achieving 97% accuracy on OPP-115 (web policies) and 87% on PPGDPR (mobile app policies), surpassing baselines.

ABSTRACT

Privacy policies serve as the primary conduit through which online service providers inform users about their data collection and usage procedures. However, in a bid to be comprehensive and mitigate legal risks, these policy documents are often quite verbose. In practical use, users tend to click the Agree button directly rather than reading them carefully. This practice exposes users to risks of privacy leakage and legal issues. Recently, the advent of Large Language Models (LLM) such as ChatGPT and GPT-4 has opened new possibilities for text analysis, especially for lengthy documents like privacy policies. In this study, we investigate a privacy policy text analysis framework PolicyGPT based on the LLM. This framework was tested using two datasets. The first dataset comprises of privacy policies from 115 websites, which were meticulously annotated by legal experts, categorizing each segment into one of 10 classes. The second dataset consists of privacy policies from 304 popular mobile applications, with each sentence manually annotated and classified into one of another 10 categories. Under zero-shot learning conditions, PolicyGPT demonstrated robust performance. For the first dataset, it achieved an accuracy rate of 97%, while for the second dataset, it attained an 87% accuracy rate, surpassing that of the baseline machine learning and neural network models.

Motivation & Objective

• Motivate the need to make privacy policies more accessible and actionable for users in the GDPR era.
• Propose a large-language-model–based framework (PolicyGPT) to analyze and categorize policy text.
• Evaluate PolicyGPT on two expert-annotated privacy-policy datasets (OPP-115 and PPGDPR) under zero-shot conditions.
• Compare LLM performance to traditional machine learning baselines and neural models on segment-level classification.

Proposed method

• Introduce a three-segment prefix prompt to encode category definitions and task context for the LLM.
• Adopt a two-step process: (1) formulate task content and category definitions; (2) provide the target text and prompt to an LLM (ChatGPT, GPT-4, Claude2) for classification.
• Use zero-shot prompting (with A/B tests showing few-shot prompts offer limited gains and higher token cost) to obtain category predictions.
• Preprocess privacy policies via web crawling and text extraction, segment into sentences or paragraphs, and map segments to one of ten categories per dataset.
• Evaluate results against human annotations, using accuracy as the main metric and reporting macro- and micro-averaged performance where applicable.

Experimental results

Research questions

• RQ1 Can large language models accurately classify privacy-policy segments into predefined categories without task-specific training?
• RQ2 How does zero-shot LLM performance compare to traditional baselines (e.g., SVM, LR, HMM, LSTM, BERT) on privacy-policy categorization?
• RQ3 What is the impact of dataset type (web-based OPP-115 vs mobile-app-based PPGDPR) on classification performance?
• RQ4 Is few-shot prompting advantageous for this task, or does zero-shot prompting suffice given LLM scale?

Key findings

• Zero-shot PolicyGPT achieves 97% accuracy on OPP-115 and 87% accuracy on PPGDPR for privacy-policy segment classification.
• PolicyGPT outperforms baseline models including Polisis and conventional ML/DNN approaches on the respective datasets.
• Few-shot prompts did not significantly improve accuracy and incurred higher token costs, leading to adoption of zero-shot prompting.
• Prompt design with a prefix prompt and explicit category descriptions enables the LLM to absorb semantic information about privacy categories and the target text.