We propose a distributional optimization-based framework to obtain improved confidence bounds of several risk measures than the previous methods. The risk measures cover entropic risk measure, conditional value at risk (CVaR), spectral risk measure, distortion risk measure, certainty equivalent, and rank-dependent expected utility, some of which are well-known in the risk-sensitive decision-making literature. We develop two estimation schemes based on the concentration bounds of the empirical distribution in terms of the Wasserstein or the supremum distance. Instead of adding/subtracting the confidence radius from the empirical risk measures, the proposed schemes evaluate the value of a specific transformation of the empirical distribution depending on the distance. The confidence bounds are shown to be consistently tighter than the previous bounds. We further verify the efficacy of the proposed framework by providing tighter problem-dependent regret bound for the CVaR bandit.