We investigate some basic questions about how neural networks learn and represent skills that are relevant to the problem of creating narrow AI systems.