Despite being a fundamental building block for reinforcement learning, Markov decision processes (MDPs) often suffer from ambiguity in model parameters. Robust MDPs are proposed to overcome this challenge by optimizing the worst-case performance under ambiguity. While robust models can provide reliable policies with limited data, the optimal worst-case performances are often overly conservative, and so they do not offer practical insights into the actual performance of these reliable policies. T…