Efficient Fine-Tuning of Large Language Models Using Principal Singular Values and Vectors

PiSSA optimizes a significantly reduced parameter space while achieving or surpassing the performance of full-parameter fine-tuning by representing the pre-trained model matrix as the product of two trainable matrices initialized with principal singular values and vectors, plus a residual matrix.