Demography, population density, business structure, and social culture differ across regions. Correlating these local factors with the number of coronavirus cases could provide more accurate hospitalization forecasts for local policy makers. Different from the classic epidemic models, we developed a data driven forecasting model solely based on deep learning. The model only requires historical data of confirmed cases, hospitalizations and deaths, demographic data of the states and optionally social distancing index. It is able to make predictions of hospitalizations and deaths in a time range of 1-4 weeks without any explicit assumption on the spreading model of Covid19. Our model achieved state-of-the-art performance, outperforming leading algorithms featured at CDC.