Eigenpruning: Improving Language Model Performance by Removing Singular Values from Weight Matrices

Eigenpruning is a method that removes singular values from weight matrices in large language models (LLMs) to improve their performance on specific tasks. This approach is inspired by interpretability methods that aim to automatically find subnetworks of a model that can effectively solve a given task.