We uncover a critical factuality-faithfulness trade-off in LLM hallucinations and propose SPACE, a novel framework theoretically and experimentally validated on TruthfulQA and PDTB to effectively address this issue.