We discuss techniques that can be used to learn how to improve AIs (dialogue models) by interacting with organic users ``in the wild''. Training models with organic data is challenging because such interactions include both high quality conversations and feedback, as well as adversarial and toxic behavior. We thus study techniques that enable learning from helpful teachers while avoiding learning from people who are trying to trick the model into unhelpful or toxic responses. We present BlenderB…