Efficient Orthogonal Finetuning of Large Models via Butterfly Factorization

The authors propose Orthogonal Butterfly (BOFT), a novel parameter-efficient finetuning method that leverages butterfly factorization to construct dense orthogonal matrices, enabling more efficient adaptation of large foundation models.