There is growing interest in multi-branch CNN architectures, due to the higher accuracy compared to single branch architectures under the same computation budget. With recent advances in high-performance hardware, single operator can no longer fully utilize the available parallelism. However, to perform CNN computations on a modern GPU, existing deep learning frameworks focus on optimizing intra-operator parallelization, resulting in a large gap between the peak performance and the real performa…