Transformer models, including GPT-4 and fine-tuned BERT variants, exhibit limited generalisability to out-of-distribution perturbations in mathematical reasoning tasks despite strong in-distribution performance.
