This work investigates the computational expressivity of recurrent neural language models (RLMs). The authors first establish an upper bound on the expressive power of RLMs by showing that RLMs with rational weights and unbounded computation time can simulate any deterministic probabilistic Turing machine (PTM) with rationally weighted transitions.
To do this, the authors introduce a variant of probabilistic Turing machines called Rational-valued Probabilistic Turing Machines (QPTMs), which can have an arbitrary number of rationally weighted transition functions. They show that QPTMs are strongly equivalent to probabilistic two-stack pushdown automata (2PDAs).
The authors then review the classical construction by Siegelmann and Sontag (1992) for simulating unweighted Turing machines with recurrent neural networks (RNNs). They extend this construction to the probabilistic case, showing how a rationally weighted RLM with unbounded computation time (called an εRLM) can simulate any QPTM.
As a lower bound, the authors study a second type of RLMs restricted to operate in real-time, meaning they can only perform a constant number of computational steps per symbol. They show that these real-time RLMs can simulate deterministic real-time rational PTMs.
The authors conclude that their results provide a first step towards a comprehensive characterization of the expressivity of RLMs in terms of the classes of probability measures they can represent.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Franz Nowak,... at arxiv.org 04-09-2024
https://arxiv.org/pdf/2310.12942.pdfDeeper Inquiries