In medicine, the integration of artificial intelligence (AI) and machine learning (ML) tools could lead to a paradigm shift in which human-AI collaboration becomes integrated in medical decision-making. Despite many years of enthusiasm towards these technologies, the majority of tools fail once they are deployed in the real-world, often due to failures in workflow integration and interface design. In this talk, I will share research using methods in human-computer interaction (HCI) to design and evaluate machine learning tools for real-world clinical use. Results from this work suggest that trends in explainable AI may be inappropriate for clinical environments. I will discuss paths towards designing these tools for real-world medical systems, and describe how we are using collaborations across medicine, data science, and HCI to create machine learning tools for complex medical decisions.