[Paper Review] Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance

Branislav Pecher, Ivan Srba+1 more|arXiv (Cornell University)|Feb 20, 2024

Science Education and Pedagogy4 citations

TL;DR

This paper investigates how many labelled samples are needed for specialised small language models to outperform general large language models in text classification. Using fine-tuning, prompting, in-context learning, and instruction-tuning across three datasets of varying complexity, the authors identify break-even points where specialised models match or exceed general models—typically requiring only 100–1000 labelled examples, with performance variance increasing the required data by 25%–200%.

ABSTRACT

When solving NLP tasks with limited labelled data, researchers typically either use a general large language model without further update, or use a small number of labelled samples to tune a specialised smaller model. In this work, we answer an important question -- how many labelled samples are required for the specialised small models to outperform general large models, while taking the performance variance into consideration. By observing the behaviour of fine-tuning, instruction-tuning, prompting and in-context learning on 8 language models, we identify such performance break-even points across 8 representative text classification tasks of varying characteristics. We show that the specialised models often need only few samples (on average $100$) to be on par or better than the general ones. At the same time, the number of required labels strongly depends on the dataset or task characteristics, with fine-tuning on binary datasets requiring significantly more samples. When performance variance is taken into consideration, the number of required labels increases on average by $100 - 200\%$. Finally, larger models do not consistently lead to better performance and lower variance, with 4-bit quantisation having negligible impact.

Motivation & Objective

• To determine the number of labelled samples required for specialised small language models to achieve performance parity or superiority over general large language models.
• To compare the impact of four data-efficient NLP approaches—fine-tuning, prompting, in-context learning, and instruction-tuning—across varying amounts of labelled data.
• To identify break-even points where specialised models surpass general models, accounting for performance variance.
• To assess how task complexity and result variance influence the required number of labelled examples.

Proposed method

• The study evaluates four NLP fine-tuning and prompting techniques: fine-tuning, prompt-tuning, in-context learning, and instruction-tuning on three text classification datasets of increasing complexity.
• Performance is measured across increasing training set sizes, from 1 to 1000 labelled samples, with statistical analysis to account for variance.
• Three datasets—SST-2, MRPC, and BoolQ—represent varying task complexity, enabling comparison across different NLP challenges.
• The authors use a consistent evaluation protocol across models, including large models (e.g., LLaMA-2, ChatGPT) and smaller models (e.g., Flan-T5), to ensure fair comparison.
• Results are analyzed for break-even points where specialised models surpass general models, with confidence intervals to account for variance.
• Compute efficiency is prioritized by reducing training steps and using shared model weights across prompting and instruction-tuning to limit resource use.

Figure 1: The impact of changing training dataset size is observed and the break-even points are identified, taking performance variance into consideration.

Experimental results

Research questions

• RQ1 At what number of labelled samples do specialised small language models achieve performance parity or superiority over general large language models in text classification?
• RQ2 How does the required number of labelled samples vary with task complexity across different prompting and fine-tuning approaches?
• RQ3 To what extent does performance variance increase the number of labelled samples needed to reliably outperform general models?
• RQ4 How do different fine-tuning and prompting strategies (e.g., instruction-tuning vs. fine-tuning) compare in data efficiency across varying dataset sizes?

Key findings

• Specialised small language models achieve performance parity or better than general large language models with as few as 100–1000 labelled samples, depending on task complexity.
• Performance variance increases the required number of labelled samples by 25% to 200%, highlighting the importance of robust evaluation protocols.
• Instruction-tuning consistently provides strong benefits, even with limited data, and outperforms other approaches in data efficiency.
• On more complex tasks like BoolQ, smaller models such as Flan-T5 show higher variance and require more careful prompt engineering (e.g., fewer in-context examples) to stabilize performance.
• The break-even point is not uniform: it is lower for simpler tasks (e.g., SST-2) and higher for complex tasks (e.g., BoolQ), with larger models showing more stable results on complex tasks.
• Prompting and in-context learning show lower variance across model sizes, but their performance is less sensitive to data size increases compared to fine-tuned models.

Figure 2: The impact of varying size of available labelled training samples (in logarithmic fashion) on the performance of fine-tuning, prompting, in-context learning and instruction-tuning approaches, reported using F1 macro and its standard deviation. Although the impact is dataset dependent (with