We propose a framework for solving constrained reinforcement learning which is risk-averse and exhibits a parameterized strong duality property under appropriate constraint qualifications.