We prove an ensemble of policies distilled on a diverse dataset improves generalisation in reinforcement learning.