Accurate electrical load forecasts of buildings are needed to optimize local energystorage and make use of demand-side flexibility. We study the usage of Trans-former neural networks for short-term electrical load forecasting of 370 buildingsfrom a public dataset. On some buildings Transformer neural networks give thebest forecasts, and on others multi-layer perceptrons are better. In addition, westudy whether models trained on a subset of the buildings generalize to unseenbuildings, and find that Transformer neural networks generalize better than multi-layer perceptrons and our statistical baselines.