In real world, our datasets often contain outliers. Moreover, the outliers can seriously affect the final machine learning result. Most existing algorithms for handling outliers take high time complexities (e.g. quadratic or cubic complexity). Coreset is a popular approach for compressing data so as to speed up the optimization algorithms. However, the current coreset methods cannot be easily extended to handle the case with outliers. In this paper, we propose a new variant of coreset technique, layered sampling, to deal with two fundamental robust optimization problems: k-median/means clustering with outliers and linear regression with outliers. This new coreset method is in particular suitable to speed up the iterative algorithms (which often improve the solution within a local range) for those robust optimization problems. Moreover, our method is easy to be implemented in practice. We expect that our framework of layered sampling will be applicable to other robust optimization problems.