We develop and analyze a scalable algorithm for multi-agent RL by sampling from the mean-field distribution of the agents to overcome the curse of dimensionality.