We find bigram subnetworks in Transformer language models that are critical to model performance.