[Paper Review] TransPolymer: a Transformer-based language model for polymer property predictions

Changwen Xu, Yuyang Wang+1 more|arXiv (Cornell University)|Sep 3, 2022

Machine Learning in Materials Science4 citations

TL;DR

TransPolymer is a Transformer-based language model with chemical-aware tokenization that enables accurate, data-efficient prediction of polymer properties by learning from large-scale unlabeled polymer sequences via masked language modeling. It achieves state-of-the-art performance across ten benchmark datasets, demonstrating the effectiveness of self-attention and pretraining in capturing polymer sequence dependencies.

ABSTRACT

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

Motivation & Objective

• To develop a deep learning model that enables accurate and efficient prediction of polymer properties without relying on expensive experiments or simulations.
• To address the limitations of existing models—such as RNNs, GNNs, and handcrafted fingerprints—by leveraging the sequence modeling power of Transformers.
• To explore whether self-attention mechanisms and pretraining on large unlabeled polymer sequences can improve generalization and representation learning in polymer science.
• To establish a data-driven framework for understanding structure-property relationships in polymers through natural language-inspired modeling.

Proposed method

• A custom polymer tokenizer with chemical awareness is designed to convert polymer sequences into tokens that preserve chemical semantics and connectivity.
• The model architecture is based on the standard Transformer encoder, using multi-head self-attention and feed-forward networks to capture long-range dependencies in polymer sequences.
• Positional encodings based on sinusoidal functions are used to preserve sequence order, as the model lacks recurrence.
• Pretraining is performed using Masked Language Modeling (MLM), where 15% of tokens are masked (80%), replaced with random tokens (10%), or left unchanged (10%) to learn contextual representations.
• Fine-tuning is performed with a one-layer MLP regressor head on downstream polymer property prediction tasks using AdamW optimizer and learning rate scheduling.
• A layer-wise learning rate decay (LLRD) strategy is applied to optimize fine-tuning, allowing higher learning rates for top layers and lower ones for deeper, more general layers.

Experimental results

Research questions

• RQ1 Can a Transformer-based language model effectively learn meaningful representations of polymer sequences for property prediction?
• RQ2 Does pretraining on large-scale unlabeled polymer sequences via masked language modeling improve downstream property prediction performance?
• RQ3 How does the self-attention mechanism in Transformers compare to RNNs and GNNs in modeling polymer sequences with variable degrees of polymerization?
• RQ4 To what extent does chemical-aware tokenization enhance the model's ability to capture chemical grammar and structural patterns in polymers?
• RQ5 Can the model generalize across diverse polymer classes and property types without requiring extensive retraining?

Key findings

• TransPolymer achieves state-of-the-art performance on ten diverse polymer property prediction benchmarks, outperforming prior methods including GNNs, RNNs, and fingerprint-based models.
• The model shows significant performance gains when fine-tuned after pretraining on a large unlabeled dataset, confirming the benefit of self-supervised pretraining.
• The use of masked language modeling during pretraining enables the model to learn chemical grammar and contextual dependencies, as evidenced by a steady decrease in binary cross-entropy loss from over 1 to ~0.07 over 30 epochs.
• Self-attention mechanisms are critical for modeling long-range interactions in polymer sequences, as ablation studies show performance drops when attention is removed.
• The model achieves strong generalization across different polymer types and property classes, including mechanical, thermal, and electronic properties.
• Pretraining on a larger dataset leads to improved performance, with model accuracy increasing with pretraining data size, demonstrating scalability.