We propose a novel search direction for stochastic online bilevel optimization, enabling first- and zeroth-order algorithms to achieve sublinear regret without window smoothing.