[Paper Review] Generating Synthetic Documents for Cross-Encoder Re-Rankers: A Comparative Study of ChatGPT and Human Experts

Arian Askari, Mohammad Aliannejadi+2 more|arXiv (Cornell University)|May 3, 2023

Topic Modeling4 citations

TL;DR

This study evaluates the effectiveness of using ChatGPT-generated synthetic documents versus human-generated responses for training cross-encoder re-rankers in information retrieval. It introduces the ChatGPT-RetrievalQA dataset and finds that models fine-tuned on LLM-generated data outperform human-trained models in zero-shot settings, while human-trained models remain superior in supervised settings, highlighting LLMs' potential for data augmentation in low-resource scenarios.

ABSTRACT

We investigate the usefulness of generative Large Language Models (LLMs) in generating training data for cross-encoder re-rankers in a novel direction: generating synthetic documents instead of synthetic queries. We introduce a new dataset, ChatGPT-RetrievalQA, and compare the effectiveness of models fine-tuned on LLM-generated and human-generated data. Data generated with generative LLMs can be used to augment training data, especially in domains with smaller amounts of labeled data. We build ChatGPT-RetrievalQA based on an existing dataset, human ChatGPT Comparison Corpus (HC3), consisting of public question collections with human responses and answers from ChatGPT. We fine-tune a range of cross-encoder re-rankers on either human-generated or ChatGPT-generated data. Our evaluation on MS MARCO DEV, TREC DL'19, and TREC DL'20 demonstrates that cross-encoder re-ranking models trained on ChatGPT responses are statistically significantly more effective zero-shot re-rankers than those trained on human responses. In a supervised setting, the human-trained re-rankers outperform the LLM-trained re-rankers. Our novel findings suggest that generative LLMs have high potential in generating training data for neural retrieval models. Further work is needed to determine the effect of factually wrong information in the generated responses and test our findings' generalizability with open-source LLMs. We release our data, code, and cross-encoders checkpoints for future work.

Motivation & Objective

• To investigate whether generative LLMs like ChatGPT can produce high-quality synthetic documents for training cross-encoder re-rankers.
• To compare the performance of cross-encoder models fine-tuned on ChatGPT-generated versus human-generated responses in both supervised and zero-shot settings.
• To assess the domain dependency of LLM-generated data effectiveness, particularly in specialized domains like medicine.
• To evaluate the feasibility of using LLM-generated responses to augment training data when labeled human responses are scarce.
• To release a new dataset, ChatGPT-RetrievalQA, to support future research in retrieval and LLM-based data augmentation.

Proposed method

• The authors constructed the ChatGPT-RetrievalQA dataset by extracting queries and responses from the HC3 dataset, which includes both human and ChatGPT-generated answers.
• Cross-encoder re-rankers were fine-tuned on two data splits: one using human-generated responses and another using ChatGPT-generated responses.
• The models were evaluated in both supervised and zero-shot settings on MS MARCO Dev, TREC DL’19, and TREC DL’20 benchmarks.
• BM25 was used as a baseline to compare retrieval effectiveness on human and ChatGPT-generated responses across train, validation, and test sets.
• Lexical overlap between queries and responses was analyzed to understand potential reasons for performance differences.
• A zero-shot scenario was simulated where models were tested on seen queries but unseen human-generated documents, assessing their generalization capability.

Figure 1. Our main experimental setup involves above steps.

Experimental results

Research questions

• RQ1 How does the effectiveness of cross-encoder re-rankers fine-tuned on ChatGPT-generated responses compare to those fine-tuned on human-generated responses in both supervised and zero-shot settings?
• RQ2 How does the effectiveness of using ChatGPT for generating relevant documents differ between specific and general domains?
• RQ3 To what extent does lexical overlap between queries and responses influence retrieval performance on human versus LLM-generated responses?
• RQ4 Can LLM-generated synthetic documents effectively augment training data in low-resource settings where labeled human responses are limited?

Key findings

• In the zero-shot setting, cross-encoder models fine-tuned on ChatGPT-generated responses significantly outperformed those fine-tuned on human responses on MS MARCO Dev, TREC DL’19, and TREC DL’20.
• On the MS MARCO Dev set, the NDCG@10 score for the MiniLM model fine-tuned on ChatGPT responses was 0.388, compared to 0.202 for BM25, indicating strong generalization to unseen human documents.
• In the supervised setting, human-trained re-rankers slightly outperformed LLM-trained models, with the MiniLM human model achieving a higher NDCG@10 than the MiniLM ChatGPT model.
• ChatGPT-generated responses exhibited 34.6% average lexical overlap with queries, compared to 25.5% for human responses, suggesting higher lexical consistency that may aid retrieval.
• BM25 retrieval performance was consistently lower on human-generated responses than on ChatGPT-generated responses across all splits, indicating that human responses are more challenging to match with queries.
• The study demonstrates that LLM-generated data can be a viable alternative for training effective zero-shot re-rankers, especially when labeled human data is scarce.