We introduce TransferBench, a comprehensive benchmark for evaluating ensemble-based black-box adversarial attacks under realistic scenarios, revealing limitations in surrogate model choices, robustness generalization, and query efficiency.