Deep neural networks have been demonstratedto achieve phenomenal success in many domains,and yet their inner mechanisms are not well understood. In this paper, we investigate the curvatureof image manifolds, i.e., the manifold deviationfrom being flat in its principal directions. We findthat state-of-the-art trained convolutional neuralnetworks for image classification have a characteristic curvature profile along layers: an initial steepincrease, followed by a long phase of a plateau,and follow…