We propose a method to enable elastic inference for pretrained ViTs via structured pruning. We do not require labels, generalize to models without a classification head and do not need re-training. We improve over the existing state of the art.