Efficient Cross-Architecture Transfer Learning for Low-Cost Inference Transformer Models

Cross-Architecture Transfer Learning (XATL) can significantly reduce the training time and improve the performance of Low-Cost Inference (LCI) Transformer models by directly transferring compatible weights from pre-trained Transformer models, without the need to train the LCI models from scratch.