There is growing interest in multi-branch CNN architectures, due to the higher accuracy compared to single branch architectures under the same computation budget. With recent advances in high-performance hardware, single operator can no longer fully utilize the available parallelism. However, to perform CNN computations on a modern GPU, existing deep learning frameworks focus on optimizing intra-operator parallelization, resulting in a large gap between the peak performance and the real performance. This performance gap is more severe under smaller batch sizes. In this work, we extensively study the parallelism between DL operators and propose Inter-Operator Scheduler (IOS) to automatically schedule the execution of multiple operators in parallel. IOS utilizes dynamic programming to find a scheduling policy specialized for the target hardware. IOS consistently outperforms state-of-the-art libraries (e.g., TensorRT) by 1.1 to 1.5x on modern CNN benchmarks.