[Paper Review] Automated Detection of GDPR Disclosure Requirements in Privacy Policies using Deep Active Learning

Tamjid Al Rahat, Tu Le+1 more|arXiv (Cornell University)|Nov 8, 2021

Privacy, Security, and Data Protection17 references4 citations

TL;DR

This paper proposes a deep active learning approach using a Convolutional Neural Network (CNN) to automatically classify privacy policies according to 18 GDPR disclosure requirements. Trained on a newly created dataset of 1,080 annotated privacy policies, the model achieves 89.2% accuracy and reveals that only 3% of websites fully comply with GDPR disclosure requirements, despite enforcement since 2018.

ABSTRACT

Since GDPR came into force in May 2018, companies have worked on their data practices to comply with this privacy law. In particular, since the privacy policy is the essential communication channel for users to understand and control their privacy, many companies updated their privacy policies after GDPR was enforced. However, most privacy policies are verbose, full of jargon, and vaguely describe companies' data practices and users' rights. Therefore, it is unclear if they comply with GDPR. In this paper, we create a privacy policy dataset of 1,080 websites labeled with the 18 GDPR requirements and develop a Convolutional Neural Network (CNN) based model which can classify the privacy policies with an accuracy of 89.2%. We apply our model to perform a measurement on the compliance in the privacy policies. Our results show that even after GDPR went into effect, 97% of websites still fail to comply with at least one requirement of GDPR.

Motivation & Objective

• To investigate the extent to which privacy policies of major websites comply with GDPR disclosure requirements after the regulation's enforcement in 2018.
• To address the lack of publicly available, high-quality datasets labeled with GDPR requirements for privacy policy analysis.
• To develop an automated, scalable method for classifying privacy policy segments into specific GDPR disclosure categories.
• To improve model accuracy cost-effectively using pool-based active learning, reducing labeling effort while increasing performance.
• To provide empirical evidence on the current state of GDPR compliance in privacy policies across 9,761 top websites.

Proposed method

• Constructed a privacy policy corpus of 9,761 websites using UK-based VPNs to ensure geographic diversity and data consistency.
• Identified 18 GDPR disclosure requirements through legal expert consultation and used them as 18 binary labels for annotation.
• Annotated 1,080 privacy policies using trained human annotators to create a high-quality, labeled dataset for model training.
• Developed a CNN-based text classification model to map privacy policy segments to the 18 GDPR requirement categories.
• Applied pool-based active learning to iteratively select the most informative unlabeled samples for labeling, improving model accuracy with minimal labeling effort.
• Used the final model to perform a large-scale compliance measurement across 9,761 websites, evaluating compliance per GDPR requirement.

Experimental results

Research questions

• RQ1 To what extent do privacy policies of top websites comply with the 18 GDPR disclosure requirements after GDPR’s enforcement in 2018?
• RQ2 How effective is a CNN-based model in classifying privacy policy segments into specific GDPR disclosure categories?
• RQ3 Can active learning significantly improve the performance of a GDPR compliance classifier while minimizing labeling costs?
• RQ4 Which GDPR disclosure requirements are most frequently omitted in privacy policies, and what are the compliance rates for each?
• RQ5 What is the overall distribution of compliance levels across websites, and how many fully comply with all 18 requirements?

Key findings

• The CNN-based model achieved an accuracy of 89.2% and an average F1-score of 0.88, representing a 10.8% relative improvement over the initial model through active learning.
• Only 3% of websites fully comply with all 18 GDPR disclosure requirements in their privacy policies, indicating widespread non-compliance.
• Six major GDPR requirement categories—such as profiling and automated decision-making—were disclosed in only 15.3% to 20% of privacy policies.
• The requirement to disclose information about profiling or automated decision-making was the most frequently omitted, with only 15.3% of websites disclosing it.
• A total of 97% of websites fail to comply with at least one GDPR disclosure requirement, highlighting systemic shortcomings in privacy policy transparency.
• The study identifies a significant gap between GDPR’s legal requirements and actual privacy policy disclosures, even years after enforcement.