[Paper Review] CARE: a Benchmark Suite for the Classification and Retrieval of Enzymes

Jason Yang, Ariane Mora+5 more|arXiv (Cornell University)|Jun 21, 2024

Advanced Proteomics Techniques and Applications7 citations

TL;DR

CARE introduces a benchmark suite for enzyme function, with Task 1 classification of enzyme sequences by EC number and Task 2 retrieval of EC numbers from reactions, plus a baseline model CREEP for multimodal retrieval.

ABSTRACT

Enzymes are important proteins that catalyze chemical reactions. In recent years, machine learning methods have emerged to predict enzyme function from sequence; however, there are no standardized benchmarks to evaluate these methods. We introduce CARE, a benchmark and dataset suite for the Classification And Retrieval of Enzymes (CARE). CARE centers on two tasks: (1) classification of a protein sequence by its enzyme commission (EC) number and (2) retrieval of an EC number given a chemical reaction. For each task, we design train-test splits to evaluate different kinds of out-of-distribution generalization that are relevant to real use cases. For the classification task, we provide baselines for state-of-the-art methods. Because the retrieval task has not been previously formalized, we propose a method called Contrastive Reaction-EnzymE Pretraining (CREEP) as one of the first baselines for this task and compare it to the recent method, CLIPZyme. CARE is available at https://github.com/jsunn-y/CARE/.

Motivation & Objective

• Formalize two realistic enzyme-function tasks: classification of enzyme sequences by EC number and retrieval of EC numbers given a reaction.
• Curate high-quality datasets linking protein sequences to ECs and reactions to ECs, with train-test splits for out-of-distribution generalization.
• Provide baselines for Task 1 and introduce CREEP as an initial baseline for Task 2.
• Enable benchmarking of state-of-the-art models for enzyme function prediction and retrieval across multiple modalities.

Proposed method

• Define two tasks (Task 1: enzyme sequence classification by EC; Task 2: EC retrieval from a reaction) and corresponding evaluation setups.
• Curate datasets from Swiss-Prot/UniProt for protein-EC mappings and EnzymeMap/ECReact for reaction-EC mappings.
• Design train-test splits to simulate out-of-domain generalization for both tasks (varying identity or difficulty levels).
• Baseline evaluation of existing models (e.g., CLEAN, BLASTp, ProteInfer) on Task 1.
• Propose CREEP, a multimodal contrastive pretraining approach, to align reactions (rxnfp) and proteins (ProtT5) for Task 2 retrieval, with optional textual modality (SciBERT).
• Provide an open-source benchmark resource at the CARE repository.

Figure 1: Overview of CARE. (A) Dataset format for CARE, showing examples of enzymes and their associated reactions. The EC number acts as a bridge between a protein sequence and the reactions it is likely to perform. The EC number is a hierarchical classification scheme for enzyme function with fou

Experimental results

Research questions

• RQ1 How well can models classify enzyme sequences into EC numbers when faced with out-of-domain sequences of varying similarity to the training set?
• RQ2 Can models retrieve EC numbers for unseen reactions by leveraging multimodal representations (protein sequences, reaction representations, and textual descriptions)?
• RQ3 Does incorporating multiple modalities (textual descriptions) improve retrieval performance for novel enzyme-reaction pairs?

Key findings

• State-of-the-art classifiers (e.g., CLEAN) outperform random baselines and crude BLASTp baselines on Task 1 across several splits.
• BLASTp remains competitive in some low-identity regimes, highlighting the value of sequence similarity baselines.
• Task 2 is more challenging due to open vocabulary (unseen ECs) and multi-modality; retrieval performance drops from easy to hard splits.
• CREEP provides a strong baseline for Task 2, especially when augmented with textual descriptions, and benefits from multimodal contrastive alignment.
• Across harder splits, most methods show significant room for improvement, underscoring the need for multimodal and advanced representation strategies.

Figure 2: Distribution of similarities between samples in each test set and the corresponding train set. (A) Protein sequence identity (Task 1) was measured to the closest hit in the train set using BLASTp. Sequence identity can be thought of as normalized Levenshtein distance. (B) Reaction similari