
Transformers can express surprisingly large classes of string-to-string transductions, including first-order rational, regular, and polyregular functions, which can be simulated using variants of the RASP programming language.


coremsg

transformers-as-formal-transducers-expressivity-and-limits


Transformers as Formal Transducers: Expressivity and Limits


title_rewrite


Temporal counting logic Kt[#] and its equivalent RASP variant C-RASP are the best-known lower bound on the expressivity of future-masked softmax transformer encoders.


compiling-temporal-counting-logic-into-softmax-transformers


Compiling Temporal Counting Logic into Softmax Transformers



This research paper demonstrates that specifically constructed causal Transformers can effectively learn to predict the next token in sequences generated by certain autoregressive functions, particularly linear functions and periodic sequences.


the-ability-of-causal-transformers-to-predict-the-next-token-in-specific-autoregressive-sequences


The Ability of Causal Transformers to Predict the Next Token in Specific Autoregressive Sequences
