[Paper Review] Patents Phrase to Phrase Semantic Matching Dataset

Grigor Aslanyan, Ian Wetherbee|arXiv (Cornell University)|Aug 1, 2022

Biomedical Text Mining and Ontologies4 citations

TL;DR

This paper introduces a new human-annotated, context-aware phrase-pair similarity dataset focused on technical terms in patents, with 48,548 entries annotated using granular semantic categories (e.g., synonym, hypernym, antonym) and contextual CPC codes. The dataset enables evaluation of models on challenging cases like phrase disambiguation and adversarial keyword matches, with Sentence-BERT achieving the best performance (Spearman 0.577), outperforming even patent-tuned BERT models.

ABSTRACT

There are many general purpose benchmark datasets for Semantic Textual Similarity but none of them are focused on technical concepts found in patents and scientific publications. This work aims to fill this gap by presenting a new human rated contextual phrase to phrase matching dataset. The entire dataset contains close to $50,000$ rated phrase pairs, each with a CPC (Cooperative Patent Classification) class as a context. This paper describes the dataset and some baseline models.

Motivation & Objective

• To address the lack of benchmark datasets focused on technical concepts in patents and scientific publications.
• To create a high-quality, human-annotated dataset with granular semantic ratings (e.g., synonym, hypernym, antonym) and contextual CPC classification.
• To include challenging examples such as phrase disambiguation, adversarial keyword matches, and hard negatives that challenge BERT-based models.
• To provide a public benchmark for training and evaluating semantic matching models on patent-specific language.

Proposed method

• The dataset was constructed by extracting salient noun and functional phrases from a patent corpus, filtering for high-frequency terms (≥100 patents).
• For each anchor phrase, up to four CPC classes were randomly sampled from the patent's classification codes to serve as context.
• Target phrases were generated via two methods: partial matching (shared tokens, different phrases) and masked language modeling using BERT to predict masked phrases.
• All phrases were preprocessed (lowercased, punctuation/stops removed) before human annotation.
• Two independent raters evaluated each phrase pair for semantic similarity and assigned one of five similarity levels with subcategories (e.g., hyponym, domain-related).
• Discrepancies in ratings were resolved through rater discussion, and rater-generated phrases were merged into the final dataset.

Experimental results

Research questions

• RQ1 How do existing pre-trained models perform on semantic matching tasks involving technical patent language with contextual cues?
• RQ2 To what extent do bag-of-words and general-purpose embedding models fail on patent-specific phrase pairs with shared keywords but different meanings?
• RQ3 Can a model fine-tuned on patent-specific data outperform general-purpose models on this semantic matching task?
• RQ4 How effective is a dual-tower architecture like Sentence-BERT on this specialized dataset compared to other architectures?

Key findings

• The best-performing model on the test set was Sentence-BERT (all-mpnet-base-v2), achieving a Spearman correlation of 0.577 and Pearson correlation of 0.598.
• Patent-BERT, a BERT model pre-trained on patent data, outperformed the general-purpose BERT-Large model, which scored 0.409 Spearman and 0.418 Pearson.
• Traditional bag-of-words models (GloVe, FastText, Word2Vec) performed poorly, with Pearson scores below 0.44, indicating limited effectiveness on complex semantic matching in technical text.
• The inclusion of contextual CPC codes significantly aids disambiguation, as evidenced by the model's ability to distinguish between phrases like 'acid absorption' and 'acid reflux' in the same context.
• The dataset contains 48,548 entries across 973 unique anchors, with 106 distinct CPC classes, and is publicly available on Kaggle for research use.
• The dataset includes diverse semantic relations such as hypernym (e.g., 'gasoline blend' → 'fuel blend'), hyponym ('faucet assembly' → 'water tap'), and domain-related matches, enabling fine-grained evaluation.