We analyze and compromise existing TEE-based secure inference methods by our Model Stealing attacks with Prior, and propose a novel secure inference method named LoRO, which is efficient, secure and accurate.