Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy. The inference cost of these models – which consist of hundreds of millions of parameters – however, can be prohibitively large. Hence, often in practice, LLMs are compressed before they are deployed in s…