RefLoRA improves upon LoRA by dynamically selecting the optimal low-rank factorization per step. This leads to faster and stabler convergence, as well as superior performance on various NLP tasks, while incurring negligible computational overhead.