we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the pre-trained models in a series of subspaces rather than a single one.