[Paper Review] A Novel Plagiarism Detection Approach Combining BERT-based Word Embedding, Attention-based LSTMs and an Improved Differential Evolution Algorithm

Seyed Vahid Moravvej, Seyed Jalaleddin Mousavirad+2 more|arXiv (Cornell University)|May 3, 2023

Academic integrity and plagiarism21 citations

TL;DR

Introduces BPD-IDE, a plagiarism detection model that fuses BERT-based word embeddings with attention-based BLSTMs, optimized with a clustering-enhanced differential evolution for pre-training and focal loss for class imbalance, evaluated on SNLI, MSRP, and SemEval2014 to outperform several baselines.

ABSTRACT

Detecting plagiarism involves finding similar items in two different sources. In this article, we propose a novel method for detecting plagiarism that is based on attention mechanism-based long short-term memory (LSTM) and bidirectional encoder representations from transformers (BERT) word embedding, enhanced with optimized differential evolution (DE) method for pre-training and a focal loss function for training. BERT could be included in a downstream task and fine-tuned as a task-specific BERT can be included in a downstream task and fine-tuned as a task-specific structure, while the trained BERT model is capable of detecting various linguistic characteristics. Unbalanced classification is one of the primary issues with plagiarism detection. We suggest a focal loss-based training technique that carefully learns minority class instances to solve this. Another issue that we tackle is the training phase itself, which typically employs gradient-based methods like back-propagation for the learning process and thus suffers from some drawbacks, including sensitivity to initialization. To initiate the BP process, we suggest a novel DE algorithm that makes use of a clustering-based mutation operator. Here, a winning cluster is identified for the current DE population, and a fresh updating method is used to produce potential answers. We evaluate our proposed approach on three benchmark datasets ( MSRP, SNLI, and SemEval2014) and demonstrate that it performs well when compared to both conventional and population-based methods.

Motivation & Objective

• Motivate plagiarism detection as a text similarity task and address class imbalance in detection.
• Propose a BERT-based embedding pipeline combined with attention-based BLSTMs for robust similarity scoring.
• Develop a clustering-enhanced differential evolution initialization and focal loss training to improve GP optimization and handling of imbalanced data.
• Evaluate the approach on standard datasets (MSRP, SNLI, SemEval2014) and compare with deep learning and metaheuristic baselines.

Proposed method

• Use BERT word embeddings to capture contextual semantics for sentence pairs.
• Employ two attention-based BLSTMs to encode source and suspicious sentences with a final similarity predictor.
• Introduce a clustering-based differential evolution algorithm to initialize network weights (two LSTMs, attention, and feed-forward layer) using a fitness function based on prediction error.
• Apply focal loss to mitigate class imbalance during training.
• Train in two phases: pre-training with DE-based initialization and fine-tuning for the final similarity task.

Experimental results

Research questions

• RQ1 How does BERT-based embedding with attention BLSTMs perform for plagiarism detection compared to traditional deep learning models?
• RQ2 Does a clustering-based differential evolution initialization improve optimization and final accuracy in this architecture?
• RQ3 Can focal loss effectively address class imbalance in plagiarism detection datasets?
• RQ4 How does BPD-IDE compare with other metaheuristic initializations and baseline methods on MSRP, SNLI, and SemEval2014?

Key findings

• BPD-IDE achieves top performance across SNLI, MSRP, and SemEval2014 compared to several deep-learning baselines.
• On SNLI, BPD-IDE attains accuracy 0.930, recall 0.920, precision 0.881, F-measure 0.900, and G-means 0.927.
• On MSRP, BPD-IDE attains accuracy 0.937, recall 0.961, precision 0.946, F-measure 0.953, and G-means 0.925.
• On SemEval2014, BPD-IDE attains accuracy 0.876, recall 0.884, precision 0.977, F-measure 0.928, and G-means 0.838.
• Using BPD-IDE with focal loss and DE initialization reduces error substantially versus random weights and FL variants, indicating the importance of both initialization and loss choice.
• BERT-based embeddings outperform one-hot, CBOW, Skip-gram, GloVe, and FastText in this setup on all three datasets.