
Transformers can express surprisingly large classes of string-to-string transductions, including first-order rational, regular, and polyregular functions, which can be simulated using variants of the RASP programming language.


coremsg

transformers-as-formal-transducers-expressivity-and-limits


Transformers as Formal Transducers: Expressivity and Limits


title_rewrite


Temporal counting logic Kt[#] and its equivalent RASP variant C-RASP are the best-known lower bound on the expressivity of future-masked softmax transformer encoders.


compiling-temporal-counting-logic-into-softmax-transformers


Compiling Temporal Counting Logic into Softmax Transformers
