Investigating the Number of Labelled Samples Required for Specialized Small Language Models to Outperform General Large Language Models on Text Classification Tasks

Specialized small language models obtained through fine-tuning or instruction-tuning can outperform general large language models used in zero-shot or few-shot settings with only a small number of labelled samples (10-1000), with the exact number depending on dataset characteristics and performance variance.