Transformer Models' Length Generalization Study: From Interpolation to Extrapolation

Transformer models can achieve complete length generalization on arithmetic tasks with the right attention biasing.