ModuLoRA: Efficient Finetuning of Large Language Models on Consumer GPUs

ModuLoRA proposes a memory-efficient finetuning algorithm for large language models, supporting 2-bit and 3-bit precision on consumer GPUs. The approach integrates user-specified weight quantizers with finetuning via low-rank adapters.