PEMA introduces a novel method for efficient fine-tuning of language models without requiring access to all model weights.