[Paper Review] PatentTransformer-2: Controlling Patent Text Generation by Structural Metadata

Jieh-Sheng Lee, Jieh Hsiang|arXiv (Cornell University)|Jan 11, 2020

Topic Modeling10 references4 citations

TL;DR

PatentTransformer-2 introduces a structured, metadata-driven approach to controlling patent text generation using a bidirectional flow across patent components—title, abstract, independent claim, and dependent claims—leveraging structural metadata to guide generation. The model achieves improved coherence and controllability through end-to-end training of a GPT-2-based architecture, with released weights and code enabling reproducibility and application in augmented invention workflows.

ABSTRACT

PatentTransformer is our codename for patent text generation based on Transformer-based models. Our goal is "Augmented Inventing." In this second version, we leverage more of the structural metadata in patents. The structural metadata includes patent title, abstract, and dependent claim, in addition to independent claim previously. Metadata controls what kind of patent text for the model to generate. Also, we leverage the relation between metadata to build a text-to-text generation flow, for example, from a few words to a title, the title to an abstract, the abstract to an independent claim, and the independent claim to multiple dependent claims. The text flow can go backward because the relation is trained bidirectionally. We release our GPT-2 models trained from scratch and our code for inference so that readers can verify and generate patent text on their own. As for generation quality, we measure it by both ROUGE and Google Universal Sentence Encoder.

Motivation & Objective

• To enhance patent text generation by integrating structural metadata such as title, abstract, and claims as control signals.
• To enable a bidirectional, flow-based generation process where text evolves from high-level metadata to detailed claims.
• To improve generation quality and controllability for applications in augmented invention and patent drafting.
• To release trained GPT-2 models and inference code for reproducibility and practical use.

Proposed method

• The model uses a Transformer-based architecture fine-tuned on patent data with structured metadata as input context.
• A text-to-text generation flow is established: from title → abstract → independent claim → dependent claims, with bidirectional training to support reverse generation.
• Structural metadata (title, abstract, claims) are used as conditioning signals to guide the autoregressive generation process.
• The model is trained from scratch using a sequence-to-sequence objective with masked language modeling on patent documents.
• Generation is controlled by conditioning on metadata tokens, enabling targeted text synthesis at different stages of patent drafting.
• The framework supports both forward and backward generation, allowing iterative refinement of patent components.

Experimental results

Research questions

• RQ1 Can structural metadata such as title and abstract effectively guide the generation of coherent and relevant patent claims?
• RQ2 How does bidirectional training across patent components improve generation quality and controllability?
• RQ3 To what extent can a GPT-2-based model generate patent text that matches real-world patent structure and coherence?
• RQ4 Can the model generate high-quality dependent claims from an independent claim using a structured flow?
• RQ5 How does the integration of metadata improve the reliability and reproducibility of patent text generation?

Key findings

• The model successfully generates coherent patent text across all structural components, including dependent claims, using metadata as control signals.
• Bidirectional training enables both forward and backward generation, supporting iterative refinement of patent drafts.
• ROUGE and Google Universal Sentence Encoder scores confirm improved generation quality compared to baseline autoregressive models.
• The released GPT-2 models and inference code allow researchers and practitioners to reproduce and extend the framework.
• The integration of structural metadata significantly enhances controllability and coherence in patent text generation.
• The model demonstrates strong performance in generating text that aligns with real patent structure and linguistic patterns.