[Paper Review] MeSH Suggester: A Library and System for MeSH Term Suggestion for Systematic Review Boolean Query Construction

Shuai Wang, Hang Li+1 more|arXiv (Cornell University)|Dec 18, 2022

Biomedical Text Mining and Ontologies23 references17 citations

TL;DR

This paper introduces MeSH Suggester, a Python library and web system that provides MeSH term suggestions for systematic review Boolean queries using both lexical (e.g., ATM, MetaMap) and neural methods (e.g., Atomic-BERT, Fragment-BERT, Semantic-BERT). The system improves query effectiveness by leveraging fine-tuned dense retrievers, with neural methods outperforming lexical baselines on CLEF 2017 and 2018 datasets, achieving F1 scores up to 0.0690.

ABSTRACT

Boolean query construction is often critical for medical systematic review literature search. To create an effective Boolean query, systematic review researchers typically spend weeks coming up with effective query terms and combinations. One challenge to creating an effective systematic review Boolean query is the selection of effective MeSH Terms to include in the query. In our previous work, we created neural MeSH term suggestion methods and compared them to state-of-the-art MeSH term suggestion methods. We found neural MeSH term suggestion methods to be highly effective. In this demonstration, we build upon our previous work by creating (1) a Web-based MeSH term suggestion prototype system that allows users to obtain suggestions from a number of underlying methods and (2) a Python library that implements ours and others' MeSH term suggestion methods and that is aimed at researchers who want to further investigate, create or deploy such type of methods. We describe the architecture of the web-based system and how to use it for the MeSH term suggestion task. For the Python library, we describe how the library can be used for advancing further research and experimentation, and we validate the results of the methods contained in the library on standard datasets. Our web-based prototype system is available at http://ielab-mesh-suggest.uqcloud.net, while our Python library is at https://github.com/ielab/meshsuggestlib.

Motivation & Objective

• To address the challenge of selecting effective MeSH terms for systematic review Boolean queries, which is time-consuming and requires expert knowledge.
• To develop a reusable, extensible library and web system that supports both established and novel MeSH term suggestion methods.
• To validate the effectiveness of neural and lexical methods in MeSH term suggestion using standard benchmark datasets.
• To lower the barrier for researchers to experiment with and deploy MeSH term suggestion techniques in medical literature search.

Proposed method

• The system implements six MeSH term suggestion methods: three lexical (ATM, MetaMap, UMLS) and three neural (Atomic-BERT, Fragment-BERT, Semantic-BERT).
• Neural methods use a fine-tuned dual-encoder model to embed keywords and MeSH terms, enabling dense vector similarity ranking.
• Fragment-BERT applies normalized CombSUM rank fusion across individual keyword embeddings, while Semantic-BERT groups similar keywords using word2vec before fusion.
• The web system exposes an API and interactive front-end where users input free-text queries and receive MeSH term suggestions from any method.
• The Python library supports model inference, evaluation with standard metrics (P, R, F1), and integration with datasets like CLEF 2017 and 2018.
• The library is extensible, allowing researchers to implement new methods by defining a custom search function in the NeuralSuggest class.

Experimental results

Research questions

• RQ1 Can neural MeSH term suggestion methods outperform traditional lexical methods like ATM and MetaMap in effectiveness for systematic review query construction?
• RQ2 How do different neural architectures (e.g., Atomic-BERT vs. Semantic-BERT) compare in terms of precision, recall, and F1 on standard MeSH suggestion benchmarks?
• RQ3 To what extent can a unified Python library and web interface streamline the development, evaluation, and deployment of MeSH term suggestion methods?
• RQ4 How does the performance of MeSH suggestion methods compare to original queries crafted by information specialists?
• RQ5 Can the system support extensibility for researchers to plug in new models or evaluation setups?

Key findings

• Neural methods, particularly Fragment-BERT, achieved the highest F1 score of 0.0690 on the CLEF 2018 dataset, outperforming all lexical baselines.
• Semantic-BERT achieved an F1 of 0.0526 on CLEF 2018, demonstrating the benefit of semantic grouping of keywords before suggestion.
• The best-performing neural method (Fragment-BERT) achieved a precision of 0.0388 and recall of 0.8034 on CLEF 2018, showing strong retrieval effectiveness.
• All neural methods produced F1 scores comparable to or better than the original query (F1 = 0.0415 on CLEF 2018), indicating strong performance despite being automated.
• The library and web system are fully functional and extensible, with support for model checkpoint switching and integration of new evaluation datasets.
• The system's evaluation pipeline, built with ir_measures, enables reproducible benchmarking and supports future method development.