We show that using checklists to automatically grade responses for reinforcement learning leads to improved instruction following