We propose the novel low-rank regularization methodology Q3R which enables robust pre-training of low-rank models for the first time in the literature.