Core Concepts
Specialized small language models obtained through fine-tuning or instruction-tuning can outperform general large language models used in zero-shot or few-shot settings with only a small number of labelled samples (10-1000), with the exact number depending on dataset characteristics and performance variance.
Abstract
The paper investigates the impact of the number of available labelled training samples on the performance and comparison of different approaches for data-efficient learning in NLP, including fine-tuning, prompting, in-context learning, and instruction-tuning. The goal is to determine how many labelled samples are needed for specialized small language models to outperform their general larger counterparts.
The key findings are:
Specialized small models often need only 10-1000 labelled samples to outperform general large models in zero-shot or few-shot settings, with the exact number depending on dataset characteristics. Binary datasets and tasks requiring better language understanding need more samples (up to 5000) compared to multi-class datasets (up to 100).
Instruction-tuned models provide a good balance, achieving near-best performance with only a fraction of the labelled samples required by fine-tuning.
Performance variance, especially from in-context learning and fine-tuning on few samples, has a significant impact, increasing the required number of labelled samples by 100-200% on average, and up to 1500% in specific cases.
Larger models do not consistently lead to better performance across approaches. Smaller models can sometimes outperform larger ones, especially in prompting, in-context learning, and instruction-tuning.
The paper provides recommendations on when to use general models (quick prototyping, limited budget) versus specialized models (large annotation budget, classification tasks), and highlights the importance of considering performance variance when comparing approaches.
Stats
Specialized small models often need only 10-1000 labelled samples to outperform general large models.
Binary datasets and tasks requiring better language understanding need up to 5000 labelled samples, compared to up to 100 for multi-class datasets.
Performance variance increases the required number of labelled samples by 100-200% on average, and up to 1500% in specific cases.
Quotes
"Specialized small models can outperform general large models using only small number of labelled examples."
"The number of required labelled samples is dependent on the dataset characteristics, with binary datasets and datasets that require better language understanding requiring significantly more labelled samples for the specialised models to outperform the general ones."
"Performance variance has a significant effect on the break-even points between models, increasing the number of required labelled samples by a large amount."