Увійти
ідея - Transformer expressivity
Transformers as Formal Transducers: Expressivity and Limits
Transformers can express surprisingly large classes of string-to-string transductions, including first-order rational, regular, and polyregular functions, which can be simulated using variants of the RASP programming language.
Compiling Temporal Counting Logic into Softmax Transformers
Temporal counting logic Kt[#] and its equivalent RASP variant C-RASP are the best-known lower bound on the expressivity of future-masked softmax transformer encoders.
The Ability of Causal Transformers to Predict the Next Token in Specific Autoregressive Sequences
This research paper demonstrates that specifically constructed causal Transformers can effectively learn to predict the next token in sequences generated by certain autoregressive functions, particularly linear functions and periodic sequences.
Продукти | Ресурси
© 2024 by Linnk AI