Origins of Linear Representations in Large Language Models

The author explores the origins of linear representations in large language models, demonstrating how log-odds matching and the implicit bias of gradient descent promote linear structures. The simple latent variable model used confirms the emergence of linear representations.