Improving Data Efficiency in Fine-tuning Large Language Models with SMALLTOLARGE (S2L)

SMALLTOLARGE (S2L) introduces a scalable data selection method for supervised fine-tuning, leveraging training trajectories from small models to guide data selection for larger models. It significantly improves data efficiency in specialized domains.