Continually learning new capabilities in different environments, and being ableto solve multiple complex tasks is of great importance for many robotics appli-cations. Modern reinforcement learning algorithms such as Proximal Policy Op-timization can successfully handle surprisingly difficult tasks, but are generallynot suited for multi-task or continual learning. Hypernetworks are a promisingapproach for avoiding catastrophic forgetting, and have previously been used suc-cessfully for continual model-learning in model-based RL. We propose HN-PPO,a continual model-free RL method employing a hypernetwork to learn multiplepolicies in a continual manner using PPO. We demonstrate our method on Door-Gym, and show that it is suitable for solving tasks involving complex dynamicssuch as door opening, while effectively protecting against catastrophic forgetting