Probing Dynamic Environments with Informed Policy Regularization