[Paper Review] Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

Peter Henderson, Mark Krass+5 more|arXiv (Cornell University)|Jul 1, 2022

Artificial Intelligence in Law44 citations

TL;DR

The paper introduces Pile of Law, a ~256GB open-source legal dataset to study data filtering grounded in law, and shows how to learn contextual privacy and toxicity filters from it to guide responsible LLM pretraining.

ABSTRACT

One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information. Emerging ethical approaches have attempted to filter pretraining material, but such approaches have been ad hoc and failed to take context into account. We offer an approach to filtering grounded in law, which has directly addressed the tradeoffs in filtering material. First, we gather and make available the Pile of Law, a 256GB (and growing) dataset of open-source English-language legal and administrative data, covering court opinions, contracts, administrative rules, and legislative records. Pretraining on the Pile of Law may help with legal tasks that have the promise to improve access to justice. Second, we distill the legal norms that governments have developed to constrain the inclusion of toxic or private content into actionable lessons for researchers and discuss how our dataset reflects these norms. Third, we show how the Pile of Law offers researchers the opportunity to learn such filtering rules directly from the data, providing an exciting new research direction in model-based processing.

Motivation & Objective

• Curate and open-source a ~256GB dataset of legal and administrative data (Pile of Law) to study data sanitization norms.
• Catalog how government laws and regulations shape privacy and toxicity constraints for data used in model pretraining.
• Demonstrate that implicit sanitization rules can be learned from legal data to guide filtering.
• Provide actionable insights and a framework for learning contextual privacy and toxicity standards from legal norms.

Proposed method

• Curate data from 35 sources including court opinions, contracts, statutes, regulations, and government publications.
• Characterize and discuss privacy and toxicity filtering norms grounded in U.S. legal systems and appendices.
• Train an initial BERT-large-equivalent model on Pile of Law and report baseline results (Appendix F).
• Develop case studies and learning methods to infer contextual privacy rules (e.g., pseudonymity decisions) from the dataset.
• Evaluate toxicity filtering across models and time, highlighting agreement/disagreement and contextual factors (Appendix).
• Illustrate how implicit sanitization rules can be learned to mimic legal redaction mechanisms.

Experimental results

Research questions

• RQ1 How can legal norms inform data sanitization practices for pretraining large language models?
• RQ2 Can contextual and jurisdictional privacy rules be learned directly from the Pile of Law dataset?
• RQ3 How do toxicity definitions and filters align with legal norms across time and contexts?
• RQ4 What are the limitations of existing toxicity filters when applied to legal texts, and how can learning from legal data address them?

Key findings

• The authors curate and open-source a ~256GB dataset of legal and administrative data from 35 sources to study data filtering norms.
• Pretraining on Pile of Law may help with challenging legal tasks and can yield comparable results to highly context-specific models (Appendix F).
• Pile of Law encodes privacy and toxicity norms that vary by jurisdiction and context, enabling learning of contextual privacy rules (e.g., pseudonymity) and contextual toxicity considerations.
• Toxicity filters show inter-model disagreement and sensitivity to content length and context, suggesting the need for more robust, value-aligned filtering approaches.
• The dataset enables learning context-aware filtering rules and measuring alignment with legal norms, highlighting shortcomings in current sanitization techniques and directing future research.