[Paper Review] PatentGPT: A Large Language Model for Intellectual Property

Zilong Bai, Ruiji Zhang+25 more|arXiv (Cornell University)|Apr 28, 2024

Intellectual Property and PatentsBusiness, Management and Accounting3 citations

TL;DR

This paper introduces PatentGPT, a domain-specific large language model fine-tuned on 240 billion tokens of intellectual property data using a standardized training procedure. It outperforms GPT-4 on the 2019 China Patent Agent Qualification Examination (scoring 65) and achieves comparable performance to GPT-4 with better cost-efficiency, especially via its MoE architecture on long-context tasks.

ABSTRACT

In recent years, large language models(LLMs) have attracted significant attention due to their exceptional performance across a multitude of natural language process tasks, and have been widely applied in various fields. However, the application of large language models in the Intellectual Property (IP) domain is challenging due to the strong need for specialized knowledge, privacy protection, processing of extremely long text in this field. In this technical report, we present for the first time a low-cost, standardized procedure for training IP-oriented LLMs, meeting the unique requirements of the IP domain. Using this standard process, we have trained the PatentGPT series models based on open-source pretrained models. By evaluating them on the open-source IP-oriented benchmark MOZIP, our domain-specific LLMs outperforms GPT-4, indicating the effectiveness of the proposed training procedure and the expertise of the PatentGPT models in the IP domain. Remarkably, our model surpassed GPT-4 on the 2019 China Patent Agent Qualification Examination, scoring 65 and matching human expert levels. Additionally, the PatentGPT model, which utilizes the SMoE architecture, achieves performance comparable to that of GPT-4 in the IP domain and demonstrates a better cost-performance ratio on long-text tasks, potentially serving as an alternative to GPT-4 within the IP domain.

Motivation & Objective

• To develop a low-cost, standardized training procedure for domain-specific large language models in the intellectual property (IP) domain.
• To address the unique challenges of IP applications, including specialized knowledge, privacy constraints, and processing of extremely long patent texts.
• To train and evaluate a series of IP-optimized LLMs that surpass general-purpose models like GPT-4 in domain-specific benchmarks.
• To demonstrate the effectiveness of sparse mixture-of-experts (SMoE) architecture in improving inference efficiency without sacrificing performance on long-context IP tasks.
• To establish a comprehensive benchmark, PatentBench, for evaluating LLMs in real-world IP use cases.

Proposed method

• Fine-tuned open-source LLaMA-based models (LLaMA2 13B, 70B and Mixtral 8×7B) on 240 billion tokens of curated IP data, including patents, file wrappers, and technical documents.
• Applied a standardized training pipeline comprising data preprocessing, continual pretraining, instruction tuning, and evaluation on domain-specific benchmarks.
• Employed 4-bit quantization and Text-Generation-Inference (TGI) 1.4 to measure inference efficiency and resource consumption on A100 GPUs.
• Designed the PatentGPT-1.0-MoE model using sparse mixture-of-experts to reduce memory usage and improve inference speed on long sequences.
• Evaluated models on the MOZIP benchmark and the 2019 China Patent Agent Qualification Examination to assess domain performance.
• Used SMoE architecture to reduce KV cache memory consumption, enabling better scalability for long-document comparison tasks in FTO analysis.

Figure 1: The distribution of different categories of pretraining data for PatentGPT models.

Experimental results

Research questions

• RQ1 Can a standardized, low-cost training procedure produce domain-specific LLMs that outperform general-purpose models like GPT-4 in the IP domain?
• RQ2 How does the performance of MoE-based models compare to dense models in terms of accuracy and inference efficiency on long-context IP tasks?
• RQ3 To what extent can fine-tuned, smaller models achieve performance levels comparable to larger models like GPT-4 in specialized IP tasks?
• RQ4 What is the impact of model architecture (dense vs. MoE) on GPU memory usage and response latency when processing long patent documents?
• RQ5 Can a custom benchmark like PatentBench provide a reliable and comprehensive evaluation of LLMs in real-world IP applications?

Key findings

• PatentGPT-1.0-Dense (70B parameters) outperformed GPT-4 on the 2019 China Patent Agent Qualification Examination, achieving a score of 65, matching human expert levels.
• PatentGPT models surpassed GPT-4 and ChatGPT-3.5-turbo on the open-source MOZIP benchmark, demonstrating superior performance in IP-specific NLP tasks.
• PatentGPT-1.0-MoE achieved performance comparable to GPT-4 while using significantly less GPU memory and showing lower response latency, especially on long input sequences.
• The MoE architecture reduced the increase in GPU memory usage with longer input sequences, making it more efficient for long-context applications like patent comparison.
• PatentGPT-1.0-MoE showed a better cost-performance ratio than dense models, indicating its potential as a scalable alternative to GPT-4 in IP workloads.
• The benchmark PatentBench was established as a comprehensive evaluation suite tailored to real-world IP use cases, including legal reasoning, document drafting, and prior art analysis.

Figure 2: The proportion of different types of data used in each pretraining stage compared to the total amount of the corresponding type of pretraining data.