Established a generalized BSM of state similarity between MDPs, backed by rigorously proved properties, which is applied to derive improved theoretical guarantees for policy transfer, state aggregation, and sampling-based estimation in MDPs.