Standard Glorot initialization becomes unstable when used in RNNs with long sequences, leading to exploding hidden states. To address this, we propose a simple rescaling that effectively mitigates the instability.

Sinusoidal initialization replaces random weight seeding with a deterministic, structured scheme that balances weights and neuron activations from the outset, yielding faster, more stable training and higher accuracy across diverse models.