Evaluating Long-Context Extension Methods for Large Language Models: A Controlled Study

Controlled evaluation of various long-context extension methods for large language models, highlighting the role of perplexity as a key performance indicator and the trade-offs between exact and approximate attention mechanisms.