Using theory and simulations, we show that task-irrelevant stimuli could cause long-term drift of neural representations