Emergent Abilities of Language Models and Pre-training Loss Perspective

Pre-training loss predicts language model performance on downstream tasks, revealing emergent abilities.