We propose a method for safely learning controlled stochastic dynamics from trajectories by incrementally expanding an initial safe control set using kernel-based confidence bounds, with theoretical guarantees on both safety and estimation accuracy.