Do Language Models Intentionally Prepare Information for Future Tokens?

Transformer language models do not intentionally pre-compute features for future tokens to a significant extent. Instead, they compute features that are useful for predicting the immediate next token, which then turn out to be helpful at future steps as well (the "breadcrumbs" hypothesis).