Gail Weiss

Memorization in transformer LMs is tied to pattern acquisition. It is non-trivial and happens in bursts according to shared patterns. Intriguingly, the relative memorization speed of larger and smaller models can change based on the pattern type.