[Paper Review] Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks

Samy Ateia, Udo Kruschwitz|arXiv (Cornell University)|Jul 18, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This study evaluates whether open-source LLMs can match commercial models in biomedical question answering using retrieval-augmented generation (RAG). Using few-shot prompting with 10 examples, Mixtral 8x7B outperformed zero-shot performance and matched commercial models like GPT-4 and Claude 3 Opus in the BioASQ 2024 challenge, demonstrating that open-source models can be competitive in domain-specific RAG tasks when fine-tuned with few examples.

ABSTRACT

Commercial large language models (LLMs), like OpenAI's GPT-4 powering ChatGPT and Anthropic's Claude 3 Opus, have dominated natural language processing (NLP) benchmarks across different domains. New competing Open-Source alternatives like Mixtral 8x7B or Llama 3 have emerged and seem to be closing the gap while often offering higher throughput and being less costly to use. Open-Source LLMs can also be self-hosted, which makes them interesting for enterprise and clinical use cases where sensitive data should not be processed by third parties. We participated in the 12th BioASQ challenge, which is a retrieval augmented generation (RAG) setting, and explored the performance of current GPT models Claude 3 Opus, GPT-3.5-turbo and Mixtral 8x7b with in-context learning (zero-shot, few-shot) and QLoRa fine-tuning. We also explored how additional relevant knowledge from Wikipedia added to the context-window of the LLM might improve their performance. Mixtral 8x7b was competitive in the 10-shot setting, both with and without fine-tuning, but failed to produce usable results in the zero-shot setting. QLoRa fine-tuning and Wikipedia context did not lead to measurable performance gains. Our results indicate that the performance gap between commercial and open-source models in RAG setups exists mainly in the zero-shot setting and can be closed by simply collecting few-shot examples for domain-specific use cases. The code needed to rerun these experiments is available through GitHub.

Motivation & Objective

• To assess whether open-source LLMs can achieve competitive performance compared to commercial models in domain-specific biomedical question answering.
• To investigate the impact of few-shot learning on open-source LLMs in a retrieval-augmented generation (RAG) setup.
• To evaluate whether QLoRa fine-tuning and Wikipedia-based context augmentation improve performance in biomedical QA tasks.
• To explore the feasibility of deploying open-source models in enterprise and clinical settings where data privacy is critical.
• To determine whether open-source models can close the performance gap with commercial models through practical, low-cost methods like few-shot prompting.

Proposed method

• Evaluated three LLMs: Mixtral 8x7B, GPT-3.5-turbo, and Claude 3 Opus in a BioASQ 2024 RAG setting using zero-shot and few-shot prompting.
• Applied in-context learning with 10-shot examples to improve instruction following and performance on biomedical questions.
• Used QLoRA fine-tuning to adapt Mixtral 8x7B and GPT-3.5-turbo on domain-specific biomedical data.
• Augmented the context window with relevant Wikipedia snippets to provide additional knowledge for the models.
• Evaluated performance on BioASQ Task B, which involves retrieving relevant PubMed snippets and generating concise, fact-based answers.
• Compared results across models and settings using standard biomedical QA metrics, focusing on answer quality and factual consistency.

Experimental results

Research questions

• RQ1 Can open-source LLMs like Mixtral 8x7B achieve performance comparable to commercial models such as GPT-4 and Claude 3 Opus in a biomedical RAG setting?
• RQ2 Does few-shot prompting significantly improve the performance of open-source LLMs in biomedical question answering compared to zero-shot inference?
• RQ3 Does QLoRA fine-tuning lead to measurable improvements in answer quality and factual consistency for open-source LLMs in this domain?
• RQ4 Does adding Wikipedia-derived knowledge to the context window enhance the performance of LLMs in biomedical QA tasks?
• RQ5 What is the trade-off between cost, privacy, and performance when using open-source vs. commercial LLMs in clinical or enterprise applications?

Key findings

• Mixtral 8x7B achieved competitive performance with commercial models like GPT-4 and Claude 3 Opus in the 10-shot setting, outperforming its zero-shot performance.
• The zero-shot performance of Mixtral 8x7B was insufficient for practical use, particularly when structured outputs were required.
• QLoRA fine-tuning did not yield consistent performance improvements for either Mixtral 8x7B or GPT-3.5-turbo in the tested setup.
• Augmenting the context with Wikipedia knowledge did not lead to measurable gains in answer quality or retrieval accuracy.
• Commercial models hosted via API were significantly faster and at least 30 times cheaper than comparable open-source inference setups.
• Few-shot prompting emerged as the most effective method to close the performance gap between open-source and commercial models in this domain-specific RAG task.