[Paper Review] Privacy Policy Question Answering Assistant: A Query-Guided Extractive Summarization Approach

Moniba Keymanesh, Micha Elsner+1 more|arXiv (Cornell University)|Sep 29, 2021

Topic Modeling37 references4 citations

TL;DR

This paper proposes a query-guided extractive summarization pipeline for privacy policy question answering that uses query paraphrasing and fine-tuned transformer models to align user queries with legal policy language. The system achieves 89% coverage in retrieving relevant answers from the PrivacyQA dataset by combining relevance and informativeness scoring on paraphrased queries.

ABSTRACT

Existing work on making privacy policies accessible has explored new presentation forms such as color-coding based on the risk factors or summarization to assist users with conscious agreement. To facilitate a more personalized interaction with the policies, in this work, we propose an automated privacy policy question answering assistant that extracts a summary in response to the input user query. This is a challenging task because users articulate their privacy-related questions in a very different language than the legal language of the policy, making it difficult for the system to understand their inquiry. Moreover, existing annotated data in this domain are limited. We address these problems by paraphrasing to bring the style and language of the user's question closer to the language of privacy policies. Our content scoring module uses the existing in-domain data to find relevant information in the policy and incorporates it in a summary. Our pipeline is able to find an answer for 89% of the user queries in the privacyQA dataset.

Motivation & Objective

• To address the challenge of semantic and stylistic mismatch between user queries and the legal language of privacy policies.
• To improve answer retrieval in low-resource settings by leveraging limited in-domain annotated data.
• To enable personalized, query-specific summarization of privacy policies for better user comprehension.
• To evaluate the effectiveness of query expansion and content scoring in enhancing answerability for privacy policy QA.

Proposed method

• The pipeline uses lexical substitution and back-translation to generate paraphrased versions of user queries, aligning their language with privacy policy text.
• A fine-tuned transformer model computes two scores—relevance and informativeness—for each policy segment relative to the expanded queries.
• The scores are combined into an answerability score to rank policy segments for summarization.
• The system retrieves and presents the top-ranked segments as a summary in response to each user query.
• The approach is evaluated on the PrivacyQA dataset using retrieval metrics such as F@k, MRR, and precision.
• An ablation study isolates the contributions of query expansion and the answer detector component.

Experimental results

Research questions

• RQ1 Can query paraphrasing improve retrieval performance in privacy policy question answering despite domain-specific language differences?
• RQ2 How effective is a hybrid scoring approach combining relevance and informativeness in identifying answer-relevant policy segments?
• RQ3 To what extent does using in-domain fine-tuning improve performance compared to out-of-domain methods?
• RQ4 What is the impact of query expansion and answer detection on retrieval recall and precision?

Key findings

• The system successfully retrieves a relevant answer for 89% of queries in the PrivacyQA dataset when considering the top 10 results.
• The model achieves an MRR of 0.59, indicating that the first relevant segment is, on average, ranked in the second position or better.
• F@5 is 80.6%, meaning 80.6% of queries have at least one relevant snippet in the top 5 results.
• The query expansion module slightly improves performance, with a 0.4% increase in F@5 and 1.1% in F@10 compared to the baseline without expansion.
• Removing the answer detector leads to a drop in F@5 and F@10 but improves precision and MRR, indicating trade-offs between recall and ranking quality.
• Domain-inspired rule-based paraphrasing outperforms out-of-domain NMT-based methods for query alignment in this task.