A novel automaton-based constrained MDP formulation and reinforcement learning algorithm for robot control under task and safety constraints specified via temporal logic.