We prove generalization bounds for neural networks which exploit approximate low-rank structure in the weights.