Action-constrained reinforcement learning (ACRL) is a popular approach for solving safety-critical and resource-allocation related decision making problems. However, one of the major challenges in solving ACRL is to find valid actions that satisfy the constraints in each RL step. While adding a projection layer on top of the original policy network is a commonly used approach, it involves solving a mathematical program, either during training or in action execution, or both, which can result in…