We inject a semantic-similarity penalty term into decoding for autoregressive and masked-diffusion LMs, generating semantically diverse answers and, with importance reweighting, enabling consistent estimation of downstream uncertainty metrics.