Theoretical Analysis of Tokenization in Large Language Models

Tokenization is a necessary step in designing state-of-the-art language models, and this paper provides a theoretical analysis of its role in enabling transformers to model complex data distributions.