This paper investigates the potential of large-scale language models (LLMs) for use in medical decision support. We begin with a discussion of the treatment problem, a core medical decision-making task for patients that is addressed in collaboration with healthcare providers. We discuss approaches to treatment problem solving that include clinical trials and observational data within evidence-based medicine, and discuss differences with treatment problems, particularly chat problems related to imitation. We highlight how LLMs can be used to address treatment problems and some of the challenges that arise. Finally, we discuss how these challenges relate to evidence-based medicine and how this might inform future steps.