[Paper Review] Unlocking Legal Knowledge with Multi-Layered Embedding-Based Retrieval

João Alberto de Oliveira Lima|arXiv (Cornell University)|Nov 12, 2024

Artificial Intelligence in Law4 citations

TL;DR

This paper proposes a multi-layered embedding-based retrieval system for legal texts that captures semantic meaning at multiple granularities—individual articles, clauses, paragraphs, and structural groupings like titles and chapters. By leveraging hierarchical embeddings and semantic chunking, the method enhances Retrieval Augmented Generation (RAG) performance, significantly improving accuracy and relevance in legal information retrieval, especially for complex queries requiring contextual depth.

ABSTRACT

This work addresses the challenge of capturing the complexities of legal knowledge by proposing a multi-layered embedding-based retrieval method for legal and legislative texts. Creating embeddings not only for individual articles but also for their components (paragraphs, clauses) and structural groupings (books, titles, chapters, etc), we seek to capture the subtleties of legal information through the use of dense vectors of embeddings, representing it at varying levels of granularity. Our method meets various information needs by allowing the Retrieval Augmented Generation system to provide accurate responses, whether for specific segments or entire sections, tailored to the user's query. We explore the concepts of aboutness, semantic chunking, and inherent hierarchy within legal texts, arguing that this method enhances the legal information retrieval. Despite the focus being on Brazil's legislative methods and the Brazilian Constitution, which follow a civil law tradition, our findings should in principle be applicable across different legal systems, including those adhering to common law traditions. Furthermore, the principles of the proposed method extend beyond the legal domain, offering valuable insights for organizing and retrieving information in any field characterized by information encoded in hierarchical text.

Motivation & Objective

• To address the limitations of keyword-based search in complex legal corpora, particularly in civil law systems like Brazil’s.
• To improve legal information retrieval by modeling semantic relationships across multiple levels of granularity in legislative texts.
• To enhance Retrieval Augmented Generation (RAG) systems by enabling context-aware, multi-granular retrieval of legal content.
• To explore the role of aboutness, semantic chunking, and inherent hierarchy in legal text representation for improved retrieval.
• To demonstrate the generalizability of the method beyond Brazilian law to other legal systems and hierarchical text domains.

Proposed method

• The method generates dense vector embeddings for legal text components at multiple levels: individual articles, clauses, paragraphs, and structural groupings (e.g., books, titles, chapters).
• It employs semantic chunking to segment long texts into semantically coherent units before embedding, preserving contextual integrity.
• Embeddings are computed using a transformer-based model, with hierarchical aggregation via arithmetic mean to represent larger sections.
• The system uses aboutness modeling to align query semantics with relevant text segments across different granularities.
• Retrieval is performed using vector similarity (e.g., cosine similarity) between query embeddings and document chunk embeddings.
• The retrieved chunks are then used as context for a LLM-based RAG pipeline to generate precise, contextually grounded legal responses.

Experimental results

Research questions

• RQ1 How can multi-granular embeddings improve the accuracy and relevance of legal information retrieval?
• RQ2 To what extent does hierarchical semantic chunking enhance retrieval performance for complex legal queries?
• RQ3 Can aboutness-aware retrieval improve the alignment between user queries and relevant legal text segments?
• RQ4 How does multi-layered retrieval compare to flat, single-granularity embedding approaches in legal RAG systems?
• RQ5 To what extent is the proposed method transferable across different legal systems and hierarchical text domains?

Key findings

• The multi-layered approach achieved higher retrieval relevance, with the multi-layer RAG answer referencing both Article 5, Inciso LXXIV and Article 134 of the Brazilian Constitution, while the flat approach omitted the foundational right in Article 5.
• The multi-layer method retrieved 7 essential chunks (E) with similarity scores ≥ 0.367 for Q8, including the critical Article 5, caput, Inciso LXXIV (similarity: 0.367759), which was missed by the flat method.
• The flat method retrieved only 8 chunks, with the most relevant being Article 134 (similarity: 0.367547), but failed to include the foundational right from Article 5, indicating a gap in semantic coverage.
• The multi-layer approach demonstrated superior contextual completeness, as evidenced by the inclusion of both the right to free legal aid (Art. 5, LXXIV) and the institutional framework (Art. 134), resulting in a more comprehensive answer.
• The system’s hierarchical structure enabled retrieval of both specific provisions and broader institutional frameworks, supporting complex queries requiring multi-level understanding.
• The method’s design supports generalization beyond Brazilian civil law, with potential applicability to common law systems and other hierarchical text domains.