[Paper Review] Measuring Patent Claim Generation by Span Relevancy

Jieh-Sheng Lee, Jieh Hsiang|arXiv (Cornell University)|Aug 26, 2019

Topic Modeling6 references5 citations

TL;DR

This paper proposes a novel span-based framework to quantitatively measure the quality of patent claim generation using span relevancy, treating claim relevance as a natural language inference problem. By fine-tuning BERT to classify span pairs for relevancy and GPT-2 to generate claims, the authors demonstrate that span relevancy decreases with higher generation diversity, validating their metric as a reliable quantitative measure for 'augmented inventing' in AI-assisted patent creation.

ABSTRACT

Our goal of patent claim generation is to realize "augmented inventing" for inventors by leveraging latest Deep Learning techniques. We envision the possibility of building an "auto-complete" function for inventors to conceive better inventions in the era of artificial intelligence. In order to generate patent claims with good quality, a fundamental question is how to measure it. We tackle the problem from a perspective of claim span relevancy. Patent claim language was rarely explored in the NLP field. It is unique in its own way and contains rich explicit and implicit human annotations. In this work, we propose a span-based approach and a generic framework to measure patent claim generation quantitatively. In order to study the effectiveness of patent claim generation, we define a metric to measure whether two consecutive spans in a generated patent claims are relevant. We treat such relevancy measurement as a span-pair classification problem, following the concept of natural language inference. Technically, the span-pair classifier is implemented by fine-tuning a pre-trained language model. The patent claim generation is implemented by fine-tuning the other pre-trained model. Specifically, we fine-tune a pre-trained Google BERT model to measure the patent claim spans generated by a fine-tuned OpenAI GPT-2 model. In this way, we re-use two of the state-of-the-art pre-trained models in the NLP field. Our result shows the effectiveness of the span-pair classifier after fine-tuning the pre-trained model. It further validates the quantitative metric of span relevancy in patent claim generation. Particularly, we found that the span relevancy ratio measured by BERT becomes lower when the diversity in GPT-2 text generation becomes higher.

Motivation & Objective

• To address the lack of quantitative metrics for evaluating patent claim generation quality in NLP.
• To enable 'augmented inventing' by developing an AI-assisted auto-complete system for patent claims.
• To model patent claim language as a span-pair relevance task using natural language inference principles.
• To validate a new metric for claim generation quality using pre-trained language models.
• To analyze the relationship between generation diversity and claim coherence in patent text.

Proposed method

• Formulate patent claim generation evaluation as a span-pair classification task, modeling relevance between consecutive claim spans.
• Fine-tune a pre-trained BERT model to classify whether two consecutive spans in a patent claim are semantically relevant.
• Use a separate fine-tuned OpenAI GPT-2 model to generate patent claims for evaluation.
• Train the span-pair classifier on labeled span pairs derived from real patent claims to simulate claim coherence.
• Apply transfer learning by leveraging state-of-the-art pre-trained models (BERT and GPT-2) for both claim generation and relevancy scoring.
• Measure span relevancy ratio as a quantitative metric of claim quality, reflecting coherence across generated claims.

Experimental results

Research questions

• RQ1 How can patent claim generation quality be measured quantitatively using span-level relevance?
• RQ2 To what extent does increased diversity in GPT-2 text generation affect the coherence of generated patent claims?
• RQ3 Can a fine-tuned BERT model effectively classify span relevancy in patent claim text?
• RQ4 Is span relevancy a reliable proxy for overall claim quality in patent generation tasks?
• RQ5 How does the structure of patent claims, with its explicit and implicit annotations, support span-based evaluation?

Key findings

• The fine-tuned BERT model effectively learns to classify span relevancy in patent claims, demonstrating the feasibility of the proposed metric.
• The span relevancy ratio decreases as the diversity of GPT-2-generated claims increases, indicating a trade-off between creativity and coherence.
• The proposed span-pair classification framework provides a reliable, quantitative measure for evaluating patent claim generation quality.
• Patent claim language contains sufficient structural and semantic cues to support span-based evaluation using pre-trained language models.
• The results validate that span relevancy is a meaningful metric for assessing the coherence and quality of AI-generated patent claims.
• The framework enables objective, automated evaluation of patent claim generation, supporting the development of AI-assisted invention tools.