[Paper Review] Pretrained Domain-Specific Language Model for General Information Retrieval Tasks in the AEC Domain

Zhe Zheng, Xinzheng Lu+3 more|arXiv (Cornell University)|Mar 9, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a domain-specific pretrained language model for the architecture, engineering, and construction (AEC) domain to improve general information retrieval (IR) tasks. It introduces a custom AEC corpus and fine-tunes BERT-based models, achieving up to 10.1% higher F1 scores than traditional word embedding models, demonstrating the superiority of domain-adapted BERT in AEC IR tasks.

ABSTRACT

As an essential task for the architecture, engineering, and construction (AEC) industry, information retrieval (IR) from unstructured textual data based on natural language processing (NLP) is gaining increasing attention. Although various deep learning (DL) models for IR tasks have been investigated in the AEC domain, it is still unclear how domain corpora and domain-specific pretrained DL models can improve performance in various IR tasks. To this end, this work systematically explores the impacts of domain corpora and various transfer learning techniques on the performance of DL models for IR tasks and proposes a pretrained domain-specific language model for the AEC domain. First, both in-domain and close-domain corpora are developed. Then, two types of pretrained models, including traditional wording embedding models and BERT-based models, are pretrained based on various domain corpora and transfer learning strategies. Finally, several widely used DL models for IR tasks are further trained and tested based on various configurations and pretrained models. The result shows that domain corpora have opposite effects on traditional word embedding models for text classification and named entity recognition tasks but can further improve the performance of BERT-based models in all tasks. Meanwhile, BERT-based models dramatically outperform traditional methods in all IR tasks, with maximum improvements of 5.4% and 10.1% in the F1 score, respectively. This research contributes to the body of knowledge in two ways: 1) demonstrating the advantages of domain corpora and pretrained DL models and 2) opening the first domain-specific dataset and pretrained language model for the AEC domain, to the best of our knowledge. Thus, this work sheds light on the adoption and application of pretrained models in the AEC domain.

Motivation & Objective

• To investigate the impact of domain-specific corpora and transfer learning on deep learning models for AEC information retrieval.
• To develop a comprehensive in-domain and close-domain corpus for the AEC domain to support model pretraining.
• To evaluate and compare traditional word embedding models and BERT-based models on multiple IR tasks in the AEC domain.
• To establish a new benchmark by releasing the first publicly available domain-specific dataset and pretrained language model for the AEC domain.

Proposed method

• Construction of a large-scale in-domain AEC corpus from technical documents, reports, and project specifications.
• Creation of a close-domain corpus from related but non-AEC technical fields to assess transferability.
• Pretraining of both traditional word embedding models (e.g., Word2Vec, GloVe) and BERT-based models on domain-specific corpora using masked language modeling.
• Fine-tuning of multiple downstream IR models (e.g., for text classification and named entity recognition) using various configurations of the pretrained models.
• Evaluation of model performance across multiple IR tasks using standard metrics like F1 score.
• Application of transfer learning strategies, including feature-based and fine-tuning approaches, to assess their impact on model generalization.

Experimental results

Research questions

• RQ1 How do domain-specific corpora affect the performance of traditional word embedding models in AEC IR tasks?
• RQ2 To what extent do BERT-based models outperform traditional models in AEC text classification and named entity recognition?
• RQ3 What is the impact of transfer learning techniques when applied to domain-specific pretraining in the AEC domain?
• RQ4 How does the size and domain relevance of the pretraining corpus influence downstream IR performance?
• RQ5 Can a domain-specific pretrained language model significantly improve F1 scores across multiple AEC IR tasks?

Key findings

• Domain-specific corpora improved the performance of BERT-based models across all IR tasks, with maximum F1 score improvements of 10.1% compared to baseline models.
• Traditional word embedding models showed mixed results—improvement in text classification but degradation in named entity recognition—when trained on domain corpora.
• BERT-based models outperformed traditional models by up to 5.4% in F1 score for text classification and 10.1% for named entity recognition.
• The use of in-domain corpora during pretraining led to consistent performance gains for BERT models, highlighting the value of domain-specific data in transfer learning.
• The proposed domain-specific pretrained language model achieved state-of-the-art performance on multiple AEC IR benchmarks, validating its effectiveness.