[Paper Review] Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning

Hamza Harkous, Kassem Fawaz+4 more|arXiv (Cornell University)|Feb 7, 2018

Privacy-Preserving Technologies in Data174 citations

TL;DR

Polisis introduces a privacy-policy analysis framework with a hierarchical multi-label CNN classifier trained on 130K policies to enable structured queries and a free-form QA system PriBot, achieving high icon and QA accuracy.

ABSTRACT

Privacy policies are the primary channel through which companies inform users about their data collection and sharing practices. These policies are often long and difficult to comprehend. Short notices based on information extracted from privacy policies have been shown to be useful but face a significant scalability hurdle, given the number of policies and their evolution over time. Companies, users, researchers, and regulators still lack usable and scalable tools to cope with the breadth and depth of privacy policies. To address these hurdles, we propose an automated framework for privacy policy analysis (Polisis). It enables scalable, dynamic, and multi-dimensional queries on natural language privacy policies. At the core of Polisis is a privacy-centric language model, built with 130K privacy policies, and a novel hierarchy of neural-network classifiers that accounts for both high-level aspects and fine-grained details of privacy practices. We demonstrate Polisis' modularity and utility with two applications supporting structured and free-form querying. The structured querying application is the automated assignment of privacy icons from privacy policies. With Polisis, we can achieve an accuracy of 88.4% on this task. The second application, PriBot, is the first freeform question-answering system for privacy policies. We show that PriBot can produce a correct answer among its top-3 results for 82% of the test questions. Using an MTurk user study with 700 participants, we show that at least one of PriBot's top-3 answers is relevant to users for 89% of the test questions.

Motivation & Objective

• Automate fine-grained annotation of privacy policies to enable scalable, multi-dimensional querying.
• Leverage a privacy-specific language model and neural classifiers to map segments to high-level and fine-grained privacy classes.
• Demonstrate applications for structured querying (privacy icons) and free-form QA (PriBot).
• Evaluate accuracy against expert annotations and user studies.
• Provide publicly accessible web services illustrating policy overview, QA, and privacy-label interfaces.

Proposed method

• Create a CorPus-based privacy-specific word embeddings (Policies Embeddings) from 130K privacy policies using fastText with subword information.
• Train a hierarchical multi-label CNN classifier to predict 10 high-level categories and 122 fine-grained attribute values per policy segment.
• Segment policies into semantically coherent fragments using HTML-based segmentation and GraphSeg with domain-specific embeddings.
• Annotate policies using the OPP-115 dataset to supervise classifiers across category and attribute levels.
• Implement an Application Layer enabling structured (predicate-based) and free-form (natural language) queries over segments and predicted classes.
• Develop PriBot, a QA system that ranks and returns relevant policy segments to user questions, evaluated against real-world questions and MTurk users.

Experimental results

Research questions

• RQ1 Can Polisis accurately assign high-level privacy categories and fine-grained attributes to policy segments?
• RQ2 How effectively can Polisis support structured queries (e.g., privacy icons) over policies?
• RQ3 Can PriBot provide relevant answers to free-form user questions about privacy practices?
• RQ4 Is the system scalable and robust across large corpora of policies?
• RQ5 How do automated icon assignments compare with expert annotations and existing certification schemes?

Key findings

• Automated privacy-icon assignment achieved an average accuracy of 88.4% across icons when compared to expert annotations.
• Category-level queries for user questions show high precision and recall, with macro-averaged precision 0.87, recall 0.83, F1 0.84, and top-1 precision 0.84.
• PriBot yields at least one correct answer in its top-3 for 82% of test questions, and 68% as the top-1 answer.
• A consumer study with 700 MTurk participants found PriBot's top-3 answers were relevant for 89% of questions.
• Table-driven evaluation of icon prediction shows varying accuracies by icon type (e.g., Automated Use 92% accuracy, Data Retention 80%, Children Privacy 98%).
• Polisis demonstrates scalability by enabling automated auditing of policy-annotation practices and icon assignment compared to traditional manual labeling.