This paper proposes a new algebraic constraint for the planar homography estimation to ensure transformations between two convex quadrilaterals. The new constraint is derived by utilizing a projective invariance of an ellipse, i.e. an ellipse is projected as an ellipse in other views under a physically plausible homography. The invariance is expressed by a quadratic inequality about a homography matrix, therefore, the quadratic constraint can be incorporated with a direct linear method that can be solved as a generalized eigenvalue problem. We demonstrate by experiments that an M-estimator with the new constraint is stable and robust against image noise and outliers compared to RANSAC family with the standard 4-point DLT method.