We introduce Gatekeeper, a novel loss function that calibrates smaller models in cascade setups to confidently handle easy tasks while deferring complex ones, significantly improving deferral performance across diverse architectures and tasks.