Language Models Implement Simple Vector Arithmetic to Solve Relational Tasks

Language models sometimes use a simple vector arithmetic mechanism to solve relational tasks by leveraging regularities encoded in their hidden representations.