[论文解读] Towards the Exploitation of LLM-based Chatbot for Providing Legal Support to Palestinian Cooperatives

Rabee Qasem, Banan Tantour+1 more|arXiv (Cornell University)|Jun 9, 2023

Artificial Intelligence in Law被引用 8

一句话总结

该论文构建了一个基于大语言模型（LLM）的聊天机器人，用以回答巴勒斯坦合作法相关问题，并通过向量化与 LlamaIndex 处理大型法律文本的方式，与专家答案进行对比，报告总体准确率为 82%，F1 分数为 79%。

ABSTRACT

With the ever-increasing utilization of natural language processing (NLP), we started to witness over the past few years a significant transformation in our interaction with legal texts. This technology has advanced the analysis and enhanced the understanding of complex legal terminology and contexts. The development of recent large language models (LLMs), particularly ChatGPT, has also introduced a revolutionary contribution to the way that legal texts can be processed and comprehended. In this paper, we present our work on a cooperative-legal question-answering LLM-based chatbot, where we developed a set of legal questions about Palestinian cooperatives, associated with their regulations and compared the auto-generated answers by the chatbot to their correspondences that are designed by a legal expert. To evaluate the proposed chatbot, we have used 50 queries generated by the legal expert and compared the answers produced by the chart to their relevance judgments. Finding demonstrated that an overall accuracy rate of 82% has been achieved when answering the queries, while exhibiting an F1 score equivalent to 79%.

研究动机与目标

• Motivate and explore how LLM-based chatbots can assist Palestinian cooperatives with legal inquiries.
• Develop a 24/7 chatbot leveraging Palestine's Law No. 20 of 2017 on Cooperatives and related bylaws.
• Evaluate the chatbot against expert-generated questions and assess accuracy, satisfaction, and bias.
• Address data scalability challenges due to large legal documents via vectorization and indexing.

提出的方法

• Construct a chatbot using ChatGPT with LlamaIndex to index and query large legal documents and a QA dataset.
• Create two Q&A datasets (human-generated and ChatGPT-generated) based on Law No. 20/2017 and related bylaws.
• Use chunking (600-token chunks, up to 8,192-token input) and 50-token overlap for vector generation.
• Evaluate the chatbot against 50 expert questions using accuracy, satisfaction, and a confusion-matrix based analysis.
• Measure performance with accuracy, average satisfaction, precision, recall, and F1 under the assumption that expert answers are correct.

实验结果

研究问题

• RQ1 Can an LLM-based chatbot accurately answer questions about Palestinian cooperative law?
• RQ2 How effective is vectorization (via LlamaIndex) in enabling large legal texts to be queried by an LLM?
• RQ3 What are the strengths and limitations of an LLM chatbot in providing legal guidance to cooperatives?
• RQ4 What is the user satisfaction level and reliability of the chatbot when evaluated against expert answers?

主要发现

• Overall accuracy achieved: 82% (41/50 questions correct).
• F1 score for right/related class: 0.88; precision for right/related: 1.0 under their evaluation assumptions.
• Average satisfaction: 78.3% based on legal counsel scoring.
• Confusion-matrix results indicate 0 for the 'wrong' class and 0.79 recall for 'right/related'.