We develop a distributed, approximate, accelerated sketch-and-project method that scales Gaussian process inference to hundreds of millions of samples.