Artificial intelligence (AI) is rapidly entering healthcare, from sepsis prediction to image analysis to patient management. Some AI systems are developed by venture-backed start-ups, others are homegrown, and many are embedded within electronic health records (EHR) systems. They demand governance: the task of ensuring safety and effectiveness at the time of integration into clinical care and throughout the product lifecycle. AI systems, including broadly deployed systems, have shown substantial quality problems and implementation challenges despite their overall promise1. Our team includes leaders at Michigan Medicine and Duke Health with substantial on-the-ground experience developing, implementing and maintaining AI systems used in clinical practice and extensive experience supporting government actors seeking to scale the potential benefits of AI. We argue the inadequacy of an exclusive focus on centralized governance — by, for example, the Food and Drug Administration (FDA), the Office of the National Coordinator for Health Information Technology (ONC), the Centers for Medicare and Medicaid Services (CMS) or even the Federal Trade Commission. Instead, centralized governors must also coordinate and support local governance within healthcare delivery settings with varying resources and capabilities in a model of collaborative governance.