
NOLA, a novel reparameterization technique, enables efficient fine-tuning of large language models by decoupling the number of trainable parameters from the model architecture and the chosen rank, breaking the rank-one decomposition limit of existing methods like LoRA.


coremsg

compressing-large-language-models-using-a-linear-combination-of-random-basis


Compressing Large Language Models Using a Linear Combination of Random Basis


title_rewrite


HydraLoRA, an asymmetric LoRA architecture, enhances the efficiency and performance of fine-tuning large language models by leveraging a shared matrix A and multiple distinct matrices B to capture both common and task-specific knowledge.


hydralora-an-asymmetric-lora-architecture-for-efficient-fine-tuning-of-large-language-models


HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning of Large Language Models



Birbal, a Mistral-7B based model, achieved a 35% performance improvement through high-quality instruction curation.


birbal-an-efficient-7b-instruct-model-fine-tuned-with-curated-datasets


Birbal: An Efficient 7B Instruct-Model Fine-Tuned with Curated Datasets
