MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

MediSwift introduces efficient sparse pre-training for biomedical language models, achieving significant reductions in training FLOPs while maintaining high performance on specialized tasks.