Core Concepts
Temporal counting logic Kt[#] and its equivalent RASP variant C-RASP are the best-known lower bound on the expressivity of future-masked softmax transformer encoders.
Abstract
The paper introduces the temporal counting logic Kt[#] and its equivalent RASP variant C-RASP. It proves that Kt[#] and C-RASP are the tightest-known lower bound on the expressivity of future-masked softmax transformer encoders with unbounded input size.
Key highlights:
- Kt[#] and C-RASP can express a variety of regular, context-free, and non-context-free languages.
- The authors prove that all Kt[#] formulas can be compiled into softmax transformer encoders.
- They show that the previous best lower bound, FOC[+; MOD], is strictly less expressive than Kt[#].
- The paper demonstrates how C-RASP can be used to construct simple transformer decoder language models with formally specifiable behavior.
- It is also shown that transformers using fixed-precision numbers can be compiled back to Kt[#].
The paper provides a strong theoretical framework for understanding the computational power of transformers, using formal logic and programming language theory.
Stats
There are no key metrics or figures used to support the author's main arguments.
Quotes
There are no striking quotes supporting the author's key logics.