We show that learning rules that are optimized to stabilize networks also create memories as a byproduct, in linear rate networks and large recurrent spiking networks.